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ABSTRACT 

The purpose of this evaluation is to describe and 
assess the political, social, economic, and educational outcomes of 
the voucher demonstration, and the implications of these outcomes for 
public policy issues. The basic hypotheses of the Demonstration, the 
basic public policy issues, the major evaluatio.n questions to be 
considered are set forth, and an organizational framework is 
specified for the analysis. The salient problems of da'-.a analysis and 
strategies for performing the central tasks of the evaluation are 
considered. Utilizing 12 specified information categories, the report 
presents a strategy for the inspection and aggregation of evaluation 
findings appropriate to address broad issues of public policy and 
enumerates specific procedures for deriving public policy 
implications from the evaluation of demonstration outcomes and 
processes. The plan concludes with discussions of a scheme for the 
management and onjanization of the evaluation, a schedule of the work 
to be performed, and the products to be delivered to OEO. 
(Author/DN) 
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INTRODUCriON 



'riiLH repurL presents Rand^s Technical Analysis Plan Eor Lhe evalu- 
ation of the pruposed GEO Elementary Education Voucher Demonstration (EKVD) . 
The purpose of the evaluation Is to describe and assess the political, 
social, economic and educational outcomes of the voucher demonstration, 
and their Implications for issues of public policy. The Plan assumes an 
l8~nionth pre-demonst rat ion period beginning in March 1972, followed by 
five consecutive one-year demonstration periods beginning In September 
J 973 at two to five demonstration sites, with a two-year post demonstra-- 
tlon period at- each site. 

The Rand i'echnlcai Analysis Plan sets forth the basic hypotheses of 
the EICVD and Lhe basic public policy issues and major evaiuatlon ciues- 
tions to be considered, and then moves directly to the specification o"^ 
an organizational framework for the analysis. We first establish twelve 
in format Ion categorie s to organize Lhe key find lags of the analysis. The 
aim is to helj) resea^'chers and policymakers understand *:he bearing of 
these findings on major issues of public policy. We then specify 40 o_u_L~ 
come dimetis ious of concern to the evaluation, i'hese dimensions of pos- 
sible demonstration outcomes are the critical variable's, of the analysis. 
The plan for data collect ion and analysis is straightforward: We first 
define the relevant indicators whose measurement will help us to specify 
the value and cjuality of each dimension of program outcomes. We then 
detail the sources of tji e data that evaluation staff will collect on 
each indicator, and the data collectloa method s that will be employed. 
Finally, we Identify the tec hniques for analyzing this data for each in- 
dicator . 

Utilizing the information categories that had been specified ear- 
lier, the plan then presents a strategy for the inspection and a ggr ega- 
tion of e valuati on findings as appropriate to address broad issues of 
public policy, and enumerates specific procedures for der i ving public 
policy implications from the evaluation of demonstration outcomes and 
processes. Finally, the plan presents a scheme for the management and 
organization of the evaluation, and a schedule of the work to be per- 
formed and the products to be delivered to OE(K Figure a presents a 
schematic overview of the Technical Analysis Plan. 
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The Tech 11. i cal Ana.1 ysls I'lan in proHonlcul in lJu' f uilowi ng yec Lions 
of Volume 1 ; 

1. FRAMEWORK Theory of Lhe lil-lVJ), public policy l.ssncy and 
major qncsLions of the evaluation, inf ornuHion caLe^ories 
and oviLcome d imcnsi ons . 

11. DKSIGN CONSIDl'KvVl'lONS — Imp Ileal: ions lor evaluaLion design 
of I'lEVl) charaeLcr lst:ics and problems associaLetl with large- 
scale social demony LraL ions » 

111. DATA COLLECTION AND ANALYSIS ~ The salient problems of 
daLa analysis, and straLegies for performing che central 
Lasks of the evaluation. 

iV. OUTCOMES AND POLICY — A plan for reaggregat ing evaluation 
f indings in support of publi c policy conclusions . 

V. MANAGEMENT AND PLANNING — The plan for management and 
organization of evaluation tasks . 

APPENDlCi:S TO THI- TECHNICAL ANALYSIS PLAN 

Volume II presents Rand cost and price information. 

A draft technical plan for surveys to be conducted as part of this 
evaluation vr/as prepared by Field Research Corporation under contract to 
Rand, and was forwarded to OEO on February 7, 1S72, in accordance with 
RFP specifications. 



1 . FllAMEWORR 



miom ov rmi voucilhr intervkntion 

There have been many varieLies of voucher plans, and a core of bnnic 
theoretica] propositions about: the cffocts of , n voucher syst(Mii is readily 
identifiable. We set forth below what we believe to be a fair statemonL 
of tlie basic hypotheses of the KEVD, based on the study prepared by the 
Harvard Center for the Study of I'ubiic Policy, and on materiais prepared 
by GEO. The assumed chain of cause and effect relationships 1b reproseuted 
in Figure 1-1. 

The propositions are: 

Increased Choice 

1.0 The voucher arrangements will increase the choice of schools 
available to parents . 

1.1 The demonstration arrangements will provide effective incentives 
for the organization of new schools. 

1.2 The voucher arrangements will allow public school parents to 
choose among existing parochial or private schools. 

1.3 Educational vouchers will allow parents to choose among a larger 
set of public schools . 

Parental Preferences 

2.0 Parents will be able Lo receive information about: 

a. . The rules of the voucher arrangements. 

b. The programs and curricula of schools in the potential set of 
choices in sufficient depth for them to develop rational prel- 

' erences among schools in the choice set. 

2.1 Parents will have or will develop preferences among jschools rei<itod 
to variations in curriculum, teaching prat:tices, student comiu^si t i on , 
or to other differences among EPiVD schools. 

Parental Influence and Sc hool Uec jponse 

3.0 Parents' wider choice among schools will increase tiieir influence? 
on the administrative policy of the schools. 
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3.1 Wider choice, and the requirement to make a choice from among 
the set of schools, will increase parental incentives to partici- 
pate in activities tliat will heighten their influence on school 
administration policies, 

3.2 Because parental choice among schools will create organizational 
uncertainty in the year-to-year procurement of resources, school 
administrators will be motivated t|) maintain an optimum level of 
student population in order to meet resource commitments, 

3.3 To maintain a student population (resource inputs) in li e with 
resource commitments, school administrators will orient their 
programs to parental preferences, 

3.4 Because school performance measures will be more available for 
parental assessment, school administrators will be motivated to 
be more concerned about the indices parents use to measure organ- 
izational performance , 

Changes in the S c hool System 

4,0 Voucher arrangements will lead to a new method fur the distribution 
of resources among schools that will provide an incentive for more 
variety in curricula and programs among schools to meet variations 
in parental preferences. 

Improved Ac hie vement 

5.0 Increased congruence between parental preferences and school out- 
puts will improve students' cognitive and noncognitive achievement. 

5.1 Parents will be in a better position to select a school, which will 
lead to an improvement in cognitive and noncognitive educational 
outcomes . 

5.2 Schools will have an incentive to modify curricula and programs to 
meet parental preferences for their target student population, and 
this, in turn, will improve educational processes and outcomes. 

Increase in Parental Satisfaction 

6,0 The changes in the school system produced by the voucher arrangement 
will increi.se parental satisfaction with the school system. 
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6.1 The increase in the school choices avar^able to the parents will 
increase parental satisfaction. 

6.2 The increase in parental influence and control of the school's 
curricula and programs will increase parental satisfaction. 

6.3 The changes in the curricula and programs of the school will 
increase parental satisfaction. 

6.4 Improvements in student performance and achievement will increase 
parental satisfaction. 

These propositions are theoretical assumptions and none is self-evidently 
true. The EEVD seeks to generate empirical data about as many of these prop- 
ositions as possible. 

PUBLIC POLICY ISSUES AND EVALUATION QUESTIONS 

Inspection of the key propositions of the theory of the EEVD reveals an 
extensive constituency whose interests may be affected by demonstration pro- 
cesses and outcomes. Students, parents, community members and community 
leaders, educators, and public servants, both elected and appointed, all have 
a "stake" in the demonstration. In addition, organization and institutional 
interests may be deeply affected: the public education system, parochial 
schools, institutions of government, professional organizations. Finally, 
there is the elusive but important stake of the public interest, in (1) economy 
and efficiency in the expenditure of public resources; (2) the amelioration or 
reduction of social conflict; (3) the effective education of the nation's 
.children; and (4) the maintenance and improvement of fair and workable rela- 
tions between citizens and their government. 

Tha public policy issues of the EEVD — and the major questions to be 
"settled" by its evaluation — are derivative of this complex set of public and 
private interests. The policymaker may ask, broadly — 

What is the desirability of extending the voucher mechanism to 
other communities? 

This question^ though, has many parts. It asks: 

o What private, organizational, and institutional interests are 
affected by the voucher mechanism, and to what extent are they 
compatible or reconcilable? 
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o What is the nature of the several public interests in vouchers, 
and to what extent are these interests compatible with one 
another and with private interests? 

In addition, the policymaker will need to extrapolate demonstration ob- 
jectives and outcomes to more general guidance for public policy* The 
original policy question is thus further elaborated ; 

1. What is the desirability of implementing some mechanism whereby 
parents can have a more direct voice in choosing the schools their 
children attend? 

2. How should educational diversity, especially the creation of new 
schools, be encouraged by public policy, if at all? 

3. Should some form of public support for private and parochial 
schools be initiated, and if so, what form should it take? 

4. To what extent should "marketplace'' incentives be introduced into 
education, and what form, if any, should such incentives take? 

These questions imply one furt:her inquiry; 

5. To what extent are (a) vouchers, and (b) the manner in which 
vouchers were implemented in the EEVD, a necessary and sufficient 
device for the attainment of the objectives of public policy, in- 
cluding those which are the subject of questions 1-4 above? 

In turn, questions dealing with the nature of public and private inter- 
ests can now be seen to apply independently to each of the broader questions 
of policy enumerated above, and to related questions that may arise, as well 
as to the question of vouchers per se > 

These are complex and difficult issues, involving many subjective and 
normative choices in addition to empirical analysis. While ''answers" to 
such questions will probably take some form of contingent probability state- 
ments ("if , then ^"), a careful and comprehensive evaluation of the 

voucher demonstration can certainly yield information that helps to make these 



Section IV of this plan provides more detailed consideration of public 
policy issues. 
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answers better informed. Inspection of the key propositions of the theory, 
of the EEVD, and a review of the public policy questions implied by that 
theory, reveals the scope of the questions that must be addressed by the 
evaluation: 

1. What has been the effect of the demonstration on the education of 
elementary school students, especially the disadvantaged? 

2, What has been the effect of the demonstration on the available 
range of choice among educational programs? 

3* What has been the impact of the demonstration on equality of 
educational opportunity? 

4, What has been, the impact of the demonstration on the economics 
of public education? 

5, How has the demonstration affected the relationship between 
citizens and their schools? 

6, What has been the impact of the demonstration on social and polit- 
ical tensions? 

These questions are demonstration-specific ; they must be answered compre- 
hensively before the broader implications of public policy can be addressed. 
Our views regarding these broader issues must for the most part be deduced 
from what we can first learn about the consequences of the EEVD as a test of 
the social, political, and educational theories that public policy conclusions 
stem from. 

Clearly, each of these broad questions has many conceptual dimensions, 
and will require a lengthy and detailed answer. The overarching test of 
evaluation processes will be the extent to which they succeed in providing 
complete and accurate answers to these questions; this in turn has a number 
of operational implications: 

We must decide: 

1. What general categories of information are of interest* 

2. What specific outcome dimensions in each information category 
must be studied in order to answer evaluation questions. 

3, What the relevant indicators are for the study of each outcome 
dimension. 

4, What sources of data are available for each indicator* 
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5. Wliat data collection mGLhods are appropriate. 

6. Wliat data analysis teclmiques uhould be utilized. 

In the balance of this section we take up the questions of general 
information categories and specific (UjLcome dimensions (1 and 2 above); 
Section 111 provides extender! discussions of data collection and analysis, 
together witli summary tables displaying indicators for eacli outcome dimen- 
sion, data sources, data collection metliods and preferred analyLica] tech- 
niques (3-6 above), 

INFQRM/^TIQN CATEGORIES 

We have selected 12 general categories for the collection of information 
bearing on the major questions of the evaluation. Their selection is grounded 
in our estimate of the most convenient way to organize the data clusters sug- 
gested by these questions and by the theory of the EEVD, They are listed in 
Table I-l, 





Table I-l 




INFORMATION CATEGORIES 


1. 


Education results 


2. 


Attitudes of practitioners 


3. 


Programs and processes 


4. 


Attributes of new scliools 


5. 


Distribution of students 


6. 


Allocation of resources 


7. 


Financial impact 


8. 


Governance and administration 


9. 


Status of professionals 


10. 


Parent attitudes and responses 


11. 


Community attitudes and responses 


12. 


Consequences beyond demonstration area 



Each of these categories of information will bring together data and 
analyses bearing on our ability to provide answers to the major questions 
of the evaluation. . The categories thus serve an accounting function tiiat 
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will help us to manage and organize large bodies of data without losing 
sight of their relationship to the ultimate purposes of the evaluation. 
Figure 1-2 shows the relationship of information categories to major eval- 
uation questions . 

OUTCOME DIMENSIONS 

Each information category will bring together analyses of a variety 
of program outcome dimensions. Outcome dimensions of interest to the evalua- 
tion are suggested by the theory and design of the EEVD, which identify major 
outcomes and processes that may be expected, and by the major questions of 
the evaluation and the information categories, which further define the areas 
of interest to evaluators. The outcome dimensions of interest are listed in 
Table 1-2. This list is provisional in the sense that demonstration processes 
are provisional; both may change in practice. Figure 1-3 displays the rela- 
tionship of outcome dimensions across professional skill areas to each of the 
information categories of the evaluation. 

In the following section of this Plan, these outcome dimensions, and 
their associated dnta collection and analysis requirsments , are treated in 
more detail. 
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TabLe 1-2 
OUTCOME D [MENS IONS 



Po Lit Ica l/SociaL 

1- Practi tioner assessment of local schools 

2 . Educational goals of practi t Loners 

3. Practitioner's opinitm about school integration 

4. Practitioner' s opi nions about EEVD 

5. Focus and scope of policy authority 

6. Demonstration area relationship to outside agencies 

7. Allocation of decisionmaking authority 

8. Administrative organization, practice and behavior 

9 . Legal and constitutit)iial rami fi cat ions 

10. Position of professionals in community 

11. Status perquisites within school system 

12. Ethnic and SES distribution o^ students 

13. Parent judgments of educational opportunities 

14. Parent assessment of local schools 

15. Parent opinions on integration 

16. Parent assessment of EEVD 

17. Parent participatiou in education of children 

18. Parent involvement in school-related activities 

19. Parent mobility 

20. Parent involvement In EEVD options 

21. Community assessment of local schools 

22. Community attitudes toward education 

23. Community opinions on integration 

24 . Community assessment of EEVD 

25. Community attitudes on political activism 

26. Inter-g oup conflict and cooperation 

27. Political and social participation 

28. Political mobilization 

29. Voting behavior 

30. Political mobilization beyond demonstration area 
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Economlc/Cost: 

Structural changes in educational marketplace 
Behavior changes in educational suppliers 
Changes in performance of educational market 
Changes in resource allocation 
Changes in fiscal flows 

Educational 

Cognitive achievement 
Affective growth 

Educational objectives of school personnel 
Teaching plans and practices 
Sociology of the classroom 
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II, DESIGN CONS ID ERATIONS 

This section discusses considerations which have influenced the 
design of the evaluation. First, we discuss the problems of evaluating 
large-scale social demonstrations. We then identify salient character- 
istics of the EEVD, and their implications for the evaluation plan. 

PROB LEMS OF ANALYSIS IN LARGE-SCALE SOCIAL DEMONSTRATIONS 

Because large-scale social demonstrations dilifer from more rigorous 
social experiments, conventional principles of experimental design are 

•k 

not precisely applicable to their evaluation. This does not exempt such 
evaluations from conventional standards of scientific rigor, nor does it 
relax the requirement that evaluators address familia^ ^Joblems of re- 
search design, such as concept definition, threats to external and in- 
ternal validity, definitions of criteria of outcome effects, and problems 
of evidence and inference. However, it will be difficult or impossible 
to achieve "acceptable" solutions to many of these problems (in the sense 
of satisfying criteria of experimental research) , The evaluation plan 
for a large-scale social demonstration must therefore be a flexible and 
br.vadly conceived instrument that does not focus narrowly on the relation- 
ships between preselected independent and dependent variables. Five 
characteristics of such demonstrations merit particular emphasis: 

1. There is likely to be considerable divergence between the 
demonstration as implemented in practice and as explicated 
in plan or theory. Program administrators may modify their 
objective.-.5 or change their practices as new opportunities 
arise or original goals are judged to have moved out of reach. 

2, Criteria of program success are difficult to define and make 
operational. In practice, they may have to change in order to 
accommodate shifting program priorities and objectives, Kroad 
program goals such as "responsiveness," "control," or "satis- 
faction" allow for many operational specifications, and it may 



On this problem, see the references listed in Appendix A, "Evalu- 
ation Objectives and Methods: A Bibliographical Note." 
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be difficult to translate thes(i into precise evaluation 
criteria. 

3. Scientific controls are seldom av'^ilable. The pre-demonstra- 
tion period in a community may be regarded as a form of control 
on the effects of demonstration independent variables, or a 
survey of a non-demonstration community may be used to assess 
the potential "swamping" effects of concurrent historical 
events on demonstration outcomes. Nonetheless, pre-demons tration 
trends cannot be an accurate guide to outcomes that could have 
been expected in the absence of the demonstration interven<:ion. 
And it is difficult to assess the mediating effects of control 
commurity characteristics that are sui generis on 

events that could have influenced outcomes in both the contro] 
and demonstration communities. Therefore, in many cases, the 
costs of attempting to maintain controls outweigh the benefits. 

4. Key program elements — staff, site., treatment. characte;-lstics — 
ar'^. seldom standardized; they vary among communities participa- 
ting ill what is conceptually a single program. Moreover, program 
elements may vary over the time period of a demonstration in the 
same community. 

5r Unknown intervening variables can be assumed to "contamxnate'' 
interventions in a complex social setting. Attempts to con- 
struct "scientifically valid" statements about cause and effect 
relationships misapply the tools of experimental design to non- 
experimental situations . 

These characteristics of large-scale social demonstrations have the 
following implications for evaluation planning: 

1. The relevance and utility of data must be assessed in terms of 
the objectives and processes of the demonstration as it actually 
occurs, rather than exclusively in terms of preselected criteria. 

2. The evaluation must be sufficiently comprehensive in the scope of 
its interests and sufficiently fine-grained in the collection and 
organization of data as to avoid the "tunnel vision" consequences 
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of concentrating exclusiveiy on preselected variables, or re~ 
cording only gross program effects. It is vital to know v>;hat 
the demonstration actually consisted of and not merely what the 
plans were. 

3. Tf pub?.ic policy inference's are to be drawn (e.g., an assessment 
cf the likeJy outcome of program replication on a larger scale), 
the evaluation mus L include not only au analysis of demonstration 
operations and results, but also an analytical extrapolation of 
e^^sential program concepts and components to a larger scale. 

4. Theory is not an adequate guide to demonstration processes, 
relationships, or outcomes, and it cannot be used as it would be 
in a controlled experiment or a narrow-aim social program, with- 
out the risk of failing to capture critical program variables. 
Theory does however play an important role: 

o It guides the collection and organization of data by 
providing an intellectual structure that allows one to 
set initial priorities for data collection. It also 
directs the organization of data in a framework that 
relates information to potential analytical conclusions. 

o It directs attention to a plausible set of demonstration 
outcomes and causal relationships, thereby providing 
initial focus and direction for analytic efforts.. 

o It provides guidance in the formulation of the questions 
to be answered by the evaluation, 

5. In order to grasp the real nature of the intervention, historical 
description of program processes and consequences is essential, 

SALIENT CHARACTERISTICS OF THE DEMONSTRATION 

The Voucher Demonstration has multiple and diverse objectives. Over- 
all program objectives are: 

o Improved education of children, especially disadvantaged children, 
o Increased parental control over the kind of schooling their 

cliildren receive, especially parents of disadvantaged children. 
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o Increased parental satisfaction with the schooling their 

children receive, especially parents of disadvantaged children. 

These objectives are to be realized as the result of a complex chain 
of cause and effect relationsl.ips which include the following intermediate 
objectives: 

o Creation of new schools. 

o Parental choices made among a range of public and private schools. 

o New incentives for teachers and administrators to be more respon- 
sive to tlie needs of children. 

o Implementation of educational innovations and program diversity 
which are responsive to the needs of children. 

There are also subsidiary program objectives associated i^rith the 
implementation and administration of tlie demonstration and the attitudes 
and behavior of diverse individuals and collectivities in the community. 

Concepts such as "control," "satisfaction," "improvemen tV (in edu- 
cation), "diversity," "incentives," and "responsiveness" have various 
possible dimensions; these dimensions must be specified concretely, and 
indices to measure these dimensions must be defined operationally to per- 
mit rigorous assessment of program success. 

EEVD objectives include effects on knowledge, attitudes, motivations, 
and behavior, as well as on the social system. Target groups and institu- 
tions include parents, other members of the demonstration area community, 
educational officials and administrators, teachers, private educational 
entrepreneurs, schools, and community groups. 

The complexity and diversity of ;;rogram objectives does not allow the 
utilization of an evaluation plan which concentrates on the causal relation- 
ships between well-defined independent and dependent variables. A more 
open-ended evaluation design is required to capture the relevant variables 
which affect the demonstracion outcomes. Given the difficulty in inferring 
causal relationships, and the requirement for description ( what happened) 
as well as analysis ( why did it happen), the evaluation plan must provide 
for the collection and display of data at various levels of aggregation 
in order to avoid obscuring the potential relevance of intermediate and 
^ short-range program processes and outcomes in larger generalizations about 

EKJC the demonstration. 
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A ccL'ollary to the niultipio program objectives of tlie EEVD is the 
wide range and diversity among the units of analysis'. Wliere the unit of 
analysis is the individual (e.g., a community leader , a school administra- 
tor), we may be interested in attributes that are absolute (belonging 
only to the person, such as age, income, years of schooling), relational 
(dealing wd th the person's relations with others, such cs number of 
friends used as sources of information), comparative (in which the person 
is characterized by comparing him along the dimensions of a given attri- 
bute with others among whom this attribute is also distributed; e.g., 
students with the highest f.cademic achievement), or context ual (in which 
the individual is describ<id according to some property of a larger social 
unit to which he belongs; e.g., students in integrated schools). 

Collectivities may also be described according to three types of 
properties; analytical (based on data about each member), structural 
(based on data about the relations among members) , and global (based on 
information about the collectivity as a unit) . There will be many kinds 
of individuals and collectivities of interest. We will specify their 
roles in the EEVD and delineate the properties according to which they are 
to be analyzed. 

Because the EEVD has such a broad scope, the evaluation plan must be 
organized so as to provide an up-to-date display of data to be collected 
in the field. Such a display will be an essential tool for evaluation 
management and will facilitate checks on the relevance and validity of the 
data that are collected. Without this management capability, there would 
be serious risk of (1) the evaluation becoming swamped with data that add 
little to the description and analysis; (2) misallocation of data collec- 
tion and data analysis resources; and (3) confusion and misunderstanding 
in the execution of program analysis tasks. 

The evaluation plan must also provide a "decentralized" approach to 
major evaluation tasks. Specialists representing various professional 
disciplines must concentrate on the appropriate data collection and 
analysis tasks in the areas of their professional competence, and their 
findings must be bruugVit together in an interdisciplinary analysis of 
aggregate outcomes. This approach to the organization and management 

Lazarsfeld, Paul F. , "Evidence and Inference in Social Research , " 
Daedalus > Fall 1958, pp. 99-130. 
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of evaluation tasks is dLutated by the demonstration's multiple object Lves 
and units analysis. The diverse data sources, data colligation nietdods, 
and data analysis techniques will require various professional skills. 
For this reason, the outcome dimensions of interest to the analysis are 
set out in this Plan according to the required professional skills, and 
provision is made fur the eventual reaggregation of information on these 
our.come dimensions for the overall analysis of the demonstration. 

In the next section, we present the strategy for a data collection 
and analysis plan which reflects these design considerations. 
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III. DATA COLLECTION AND ANALYSIS 

INTRODUCTION 

This section discusses salient problems of data collection and analysis 
for each of the outcome dimensions of interest to the evaluation. The dis- 
cussion is organized into three subsections: 

A. Political/Social Outcome Dimensions 

B. Economic/ Cos t Outcome Dimensions 

C. Educational Outcome Dimensions 

At the end of each subsection, a table summarizes and displays the key indi- 
cators, data sources, data collection methods, and data analysis techniques 
related to each outcome dimension. 

Section IV of this Plan discusses the manner in which this information 
will be reaggregated for program-level analysis. 
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III. A. SOCIAL/POLITICAL OUTCOME DIMENSIONS 



The social and political outcomes that we will discuss in this 
subsection will be measured primarily in the context of the community, 
viewed as a social system containing, among other smaller systems, 
the school system. While one may analyze any social unit as a 'V-losed** 
system (concentrating on its internal structures and processes) , we 
have chosen to view both the demonstration community and its schools as 
open systems — allowing us to capture both internal system effects and 
relationships between systems. Indeed, this approach is necessary since 
the EEVD originates as an intervention from outside the public school 
system and local community, and is designed explicitly to alter the 
relationships between citizens and their schools* If changes occur in 
the relationships bf the schools and the community, one may expect that 
the internal relationships in each system will undergo change as well: 
citizens of different constituencies may find themselves in different 
positions and roles relative t'o one another; school personnel may find 
their accustomed positions and behavior modified. 

Three of our six evaluation issues will be addressed primarily by 
data from the community context: (1) has the demonstration restructured 
the relationship between citizens and their schools in a desirable manner? 
(2) has the demonstration helped to ameliorate or reduce critical social 
and political tensions? (3) has the demonstration increased equality of 
educational opportunity? The remaining evaluation issues will be 
addressed by v. %a from other contexts though inputs from the community 
sector will be used as appropriate. 

For the purpose of the evaluation, the significant groups in the 
community are the parents, other citizens, school personnel, and students. 
The most significant variables for analyzing the demonstration outcomes 
for each of these groups are their experiences (booadly defined to include 
knowledge), their attitudes , and their behavior . The substance of these 
major variables (that is, the specific experiences, attitudes, and behavio 
wa ohoose to inquire about) is defined by the requirements of those 
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infontjation categories that are needed to answer the major evaluation 
questions. Further, we must be able to measure the preseuce or 
absence of change, the substance of the change, and the degree of 
change in these variables over time — and the relationship of the 
change process to the intervention of the EEVD. Finally, we must 
be able to differentiate between outcomes that are intended by the 
BEVD and those that may be called latent functions or unintended 
consequences of the demonstration. 

DATA COLLECTION TECHNIQUES 

To carry out these tasks, we plan to use four techniques of data 
collection: STirveys, interviews, examination of public record^, and 
community observation. Although considerable work has been devoted to 
the development of the survey instruments and sampling design, surveys 
are not necessarily the primary form of data collection for the measure- 
ment' of social and political outcomes. (We have spent a considerable 
portion of our planning effort An developing a baseline survey instrument 
suited to the needs of the evaluation and the characteristics of the 
anticipated respondents as described in our submission of February 7, 1972, 
Survey Research Specifications and Baseline Instruments for EEVD. In 
Appendix B we discuss the problems of survey research, the results of the 
pretest of our draft baseline survey instrument, and their implications 
for changes in the final survey instrument.) 

While surveys provide the most reliable '-Jay of acquiring certain types 
of information (e.g., the attitudes of people that may '>e expressed in no 
other medium available to policymakers), they are inefficient ways of 
acquiring other equally important types of information. Suirveys may be 
used to assess the experiences and opinions of ordinary people in the 

On pages 29"30 of this subsection, we discuss revisions to the 
sampling design submitted in that report. 
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comraunity who have either marginal or no associations with channels 
of opinion formation and expression beyond their families and friends. 
Although organizational leaders are often willing to .°peak for 
these people, their responses are not representative. On the other hand 
organizational leaders provide insights into the impacts of the 
demonstration that cannot be acquired through the surveys. These are 
the people in positions that make them most able to vocalize their 
opinions in public media, influence and mobilize constituencies, and 
respond to the demonstration in organized fashion. They are also privy 
to discussions of the demonstration — and thus information — not 
accessible to ordinary citizens. We thus view the data gathered by 
means of surveys and those gathered by means of personal interviews as 
balancing devices in the analysis — each adding to analysis of the othe 

The measuroment of people's behavior represents a different type 
of problem. Since behavior represents overt action, in some instance 
it may be recorded in public records. We have tried to avoid including 
behavioral items in the survey instrument that could be acquired more 
cheaply through an examination of school, voting and other public record 
Obviously one cannot eliminate all behavioral items from the survey in- 
struments because (a) some behavior is not recorded anywhere and thus 
must be acquired in the surveys (for example, parent-child interaction 
about school) , and (b) records do not provide reasons for the behavior 
which for evaluation purposes may be as important as the behavior it- 
self. (If evidence shows that parents keep their children in accustomed 
schools because they perceive no differences between schools, the policy 
implications would be different from those produced by evidence that 
parents keep their children in accustomed scl'iools because they want 
their children in schools close to home, whatever the perceived dif- 
ferences among schools.) To be sure, respondent reasons for reported 
behavior may be consciously or unconsciously misleading. The best 
guide to ij.nalysis in these cases is a comparison of stated reason 
versus aggregate outcomes in the light of the most important policy 
questions.' For example, large numbers of respondents may express op- 
position to racially and ethnically segregated schools and may express 
reasons for school choices and school admissions policies unconnected 
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Lo racial/ethnic considerations, but schooi (inrollment records may con- 
tinue to show segregated schools. Regardless of what people say about 
their decisions, the outcomes of their decisions are likely to be more 
important lor policy purposes. 

The fourth technique for data collection is community observation, 
which will also provide important kinds of data. Community observation 
has three merits for the evaluation. First, it is the best way of ac- 
quiring an understanding of the patterns of group relationships, commu- 
nity norms, and daily activities that make up the life of the demonstra- 
tion community. "Still-life" portraits of a community are not adequate 
to capture the flow of demonstration effects and the range of affected 
groups. Community observation provides something more akir, to a moving 
picture of a community-in-process. Second, it is one of the most impor- 
tant sources of data and insight into the unintended consequences of the 
demonstration. It is precisely because we cannot predict these conse- 
quences or the groups they may affect that we cannot be certain of elicit- 
ing them from the surveys, interviews, and inspection of records. Trained 
observers who live in the community will often be able to note subtle 
changes in group participation, group concerns, community norms that 
may reflect unanticipated demonstration effects. Tliis ability is some- 
times referred to as "getting a feel" for the community — a description 
which inevitably sounds unscientific and ambiguous* However, good com- 
munity observation works by ground rules which have been developed pre- 
cisely to translate the inchoate collections of isolated observations by 
which people make judgments about the "state" of their community into 
more explicit sets of indices. In short, neighborhoods do have "climates," 
communities do have "rhythms" of daily life, and personal and group reputa- 
tions are frequently in a state of constant though subtle flux. Surveys, 
personal interviews, and official records are simply too selective to 
"catch" tVie indices necessary to understand these areas of community life. 

The third merit of community observation is its role in providing 
data which can be used to formulate schedules for more structured face- 
to-face interviews. The more one knows about the way a community is 
functioning, the better one knows whom to interview, what questions to 
ask, and what issues to probe. 
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One of the tasks in the pre-demonstration period will be to develop 
a plan for training community observers, followed by the recruitment and 
training of the observers. This is one of the most important components 
of the evaluation effort. A training program, yielding people whose data 
gathering and reporting will be comparable and of consistent high quality, 
is essential to its success. 

Our community observers will be responsible for collecting, organiz- 
ing, and reporting most of the documentary data about the community. Of- 
ficial statistics, public records, newspapers, organizational records are 
some typical sources for these data. They will also be responsible for 
interviewing community leaders and attending appropriate '^'^nimunity events 
and meetings. We plan to reci'uit and train two community ooservers for 
each site, one male and the other female. 

ANALYSIS OF ATTITUDES AND BEHAVIOR 

In the analysis of social and political outcomes, we will be constantly 
measuring attitudes against behavior within and between relevant groups 
in the demonstration community. Tlie experience of social science research 
is discouraging at first glance^ for it shows that attitudes are sometimes 
perfectly congruent with behavior, sometimes opposite to behavior, and are 
often somewhere in-between these two extremes. This means, in effect, that 
options to act: in accord with one's beliefs and/or preferences are often 
viewed as too costly. In order to compare attitudes and behavior, we must 
identify the perceived constraints that carry most weight in people's de- 
cisions to ac*: in a particular way. Multivariate analysis of our survey 
responses will be especially helpful for this task. 

The EEVD seeks to change the costs of educational options for parents, 
creating a new set of options for them. As constraints shift, formerly 
latent attitudes may become operative. This is why the baseline survey 
instrument asks parents to choose between options fox their childrens' 
education that are generally unavailable now, but should become available 
as the demonstration begins. It is also why the evaluation design provides 
for the detection of unanticipated consequences; as constraints shift in 
one area of community life, then constraints may also shift in other areas. 

ERIC 



-29- 



Furthermore, attitudes held under known constraints may appear 
stronger than when the constraints are lifted. In such cases, people 
may not "follow through'^ on their expressed interests and desires when 
the opportunity arises. Indeed, one of the most ch^'irished aspects of 
freedom may be the ability to assert strong negative feelings about 
something without feeling obliged to act on thes3 feelings. 

In short, the analysis of attitudes and behavior is never simple 
or straightforward. We have tried here to point out the major analytic 
pitfalls ind the considerations and analysis techniques we will bring 
to bear in attempting to avoid them. 

ALTERNATIVE SAJ^LING DESIGN 

The sampling design which we submitted in Survey Research Specifica- 
tions and Baseline Survey Instrument for EEVD , February 7, 1972, was de- 
veloped wir.h priority placed on quality of the T:.:sult5 rather than cost. 
Having made the cost estimates for the initial sample, however, we find 
that a disproportionate amount of the resources available for the evalua- 
tion will be spent for surveys. Thus, as we indicated in our letter of 
transmittal accompanying the sampling design and draft baseline survey 
instruments, we fire submitting an alternative sampling design that will 
reduce survey costs to a more realistic proportion of the total evaluation 
budget in each community. 

Our initial design propr^ses 1300 completed interviews for each sur- 
vey of parents and other citizens in the demons tratiovi community and 390 
completed interviews in the control community. Our alternative design 
calls for 800 completed interviews for each demonstration community sur- 
vey and no control community survey. Since we have not varied the in- 
terview design in the alternative plan, each cell size is reduced to .615 
of the original cell size (800/1300). For each cemsus tract the number 
of clusters sampled will be reduced by the same proportion. 

A sample size of 800 will still detect differences of a reasonable 
order of magnitude. For example, a random sample of 800 on the propor- 
tion of "y^s's" on a "yes«*no" question brackets the population proportion 
+ 7% with over ,95 confidence. However, to estimate sample sizes neces- 
^ sary to get specified accuracy of a prediction with multivariate regression 

EKJC 
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models requires an idea of the standard error of predictioa — impossible 
without some data. This suggests that sample size should be determined 
sequentially both within each demonstration community and between demon- 
stration communities, as information from one community can be used in 
planning for anotlier community. Wliile we have used the number 800 for 
our cost estimates, we strongly recommend that initial baseline data 
from the first demonstration community be used in determining sample 
size for subsequent survt?ys and subsequent communities. 

llie "control group'* in the initial design con.sisted of a sample 
from one tommunity near the demons t rat ion community . Each interview is 
an abbreviated version of the demonstration community instrument. This 
would allow us to monitor general attitudes toward education affected by 
national trends in education- related issues (e.g., the Serrano v. Pries t 
decision or other significant changes in the financing of public schools). 
Even so, the neighboring control community may have its own peculiarities, 
and our resulcs frcm that community would be vulnerable to challenge 
along those lines. The better way to develop "controls" would be to use 
a relatively large number (say 20) of different control communities with 
full interviews to a small sample in each community. In this way one could 
control for community differences but still detect any national or state- 
vide changes in attitudes toward education which might also affect the 
demonstration community. However, this method was rejected from the be- 
ginning because of its very high cost. 

Since the results from one control community are of qj-estionable 
value, we prefer to eliminate tho^se surveys altogether, given funding 
constraints. If there is more than 'me voucher demonstration, other 
demonstration communitiea will serve some of the purposes of the neigh- 
boring control community and may serve them even better since in these 
conimunities the full inte'iview will be administered. However, if GEO 
prefers to include a separate control community, we are prepared to con- 
duct the survey as described in the Field Research Corporation report. 

DATA SOURCES 

We intend to acquire data relevant to social and political outcomes 
from four major groups in the community: parents, other citizens, school 
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personnel, and students ► We now discuss the important classifications 
within these groups, the particular data collection devices appropriate 
to each group, and the substantive areas of interests 

Parents 

We will focus on the experiences, attitudes, and responses of five 
subgroups of parents distinguished by the nature of their relationship 
to the demonstration : 

1. Parents with preschool children 

2. Parents with children in K-n (target parents) 

3. Parents with children in grades beyond the demonstration 

4. Parents holding elective or appointive positions on parent 
groups or committees active in educational affairs 

5. Parents who use their vouchee's to choose different schools 
for their children* 

Respondents to the first three subgroups will be selected in the 
course of the random probability sampling for the full parent/community 
survey . 

Parents with pre-school children are of interest in comparing the 
perspectives of "potential" as opposed to "actual" participants in the 
demonstration* Over time, the effects of their contact with participating 
cohorts of parents will help to confirm our assessments of the response of 
participating parents. In addition, baseline data for this group helps 
correct for the error inherent in recall of the original participants 
when we need Lo compare pre-voucher attitudes with those that develop 
during the demonstration* 

Parents with school children beyond an age eligible for the voucher 
have had extensive experience with the school system* They may provide 



We are interested in parents who are members of such groups af PTA, 
executive committees, Pareiut Advisory Committees for Title I and Follow 
Through Programs, and other community groups, official or ex officio, ac- 
tive in education* As the demonstration proceeds, parents participating 
in groups formed in response to the EEVD such as EVA committee or Concerned 
Mothers group, will be included in the evaluation* 
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a source of evidence from which some inferences can be drawn about the 
extent of EEVD influence on parent attitudes and responses. Some parent 
concerns about schools may simply be a function of the child's develop- 
ment. If changes in target parents' attitudes and responses were due 
solely to maturation factors), we would expect to find their response 
becoming more and more like those of parents of older children. Such in- 
ferences must be cautious, for numerous factors occurring concurrently 
with the EEVD, such as Affirmative Action programs, might be producing 
changes in the post demonstration parents. In later surveys, these par- 
ents will have experienced the change from voucher to non-voucher school- 
ing and their response may be significant for policy considerations to 
extend vouchers to the secondary school systeiu. 

Although some parents in the fourth and fifth subgroups are likely 
to be interviewed in the surveys, we cannot get an adequate representation 
of either group by this means. Thus, these parents will be interviewed 
with a much more focused set of questions. Parent committee members have 
a singular contribution to make to the evaluation: an assessment of the 
degree to which school personnel respond to parent suggestion? and recom- 
mendations as a consequence of the demonstration. These parents who act 
as part of the "official" channels of communication should have an exper- 
tise about parent-school relationships unavailable to most parents. Rand 
personnel will attend their meetings from time to time, interview the 
officers of thes,*i groups, and inspect organizational records (minutes, 
group proposals, etc.) which they make available to us. 

If we could assume that large proportions of parents will change 
the schools which their children attend in the first year of the demon- 
stration, we could plan to select them in the course of the survey sampl- 
ing. We do not think, however, that it is wise to make this assumption. 
Thus, we plan to provide schools with a form (via the Data Management 
Contractor) which parents would fill out at the time they fill out other 
forms when changing their child's school. This form will ask them to say 
if they are willing to be interviewed by a saember of th^ evaluation team. 
We would then receive from the Data Management Contractor the names of 
parents who had changed schools and interview the volunteers, primarily 
concerning their reasons for selecting another school. Their responses 
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will cast light on EEVD impact on the most active Users of the options 
it provides . 

The most pertinent categories of information which we will be ac- 
quiring from parents, identified according to their source and functions 
in the analysis, are explicitly set forth in Table III-l at the end of 
this subsection. Here, wc want to indicate our major reasons for collect- 
ing these particular types of data. In the data specifications we have 
established priorities on the basis of the major evaluation issues and 
policy questions that must finally be addressed by the evaluation. In 
particular, it is essential to find out about parent attitudes and re- 
sponses toward: (1) the school to which they have access, (2) racial/ 
ethnic integration in the schools, and (3) the operations of the EEVD. 
We have created baseline measures for these elements which will be con- 
tinued in the demonstration surveys. 

While the information categories, outcome dimensions, and indices 
relevant to racial /ethnic integration and the operations of the EEVD are 
fairly straightforward, important distinctions must be made in parent at- 
titudes and responses toward schools. For instance, it is certainly pos-* 
sible for pare: =5 to value forroal education highly and still be antagon- 
istic to existing schools; also, attitudes toward education may change 
differently from attitudes toward schools. Indeed, Rand's advisory panel 
of experts on the EEVD has suggested that we might realistically anticipate 
thti attitudes of poor parents to grow more critical toward the schools 
precisely as they became more concerned with the education of their children. 
We have thus maintained a careful analytic distinction between attitudes 
toward schools and toward education. 

Other Citizens 

The analysis must include people who hold no official positions in 
the schools and who are unaffected by the demonstration because they 
neither have nor expect to have children in the schools. While these 
people may be less interested in certain features of the demonstration, 
as taxpayers they are part of the constituency of the school system, and 
can be mobilized around political issues covering the schools. No commu- 
nity is so structured as to prevent exchange between parents and non-parents 
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or taxpayers and non- taxpayers — and the flow of information and opinion 
between these differing groups is of considerable import. For instance, 
different opinions and responses between EEVD parents and other citizens 
may emerge. The evaluation cannot decide whose judgments should prevail, 
but the evaluation may provide data that may be helpful in making such a 
decision. 

We will be examining two subgroups of "other citizens": (1) members 
of the community selected on a random probability basis, and (2) community 
leaders — those persons who hold elective or appointive offices in various 
civic and community groups,, and those who are identified by community mem- 
bers as informal leaders. 

Those in the first category will be surveyed each year with an in- 
strument comparable to the parent survey appropriately modifieu. Their 
attitudes and responses toward education, schools, and the EEVD are import- 
ant for the evaluation. Those in the second category will be studied 
primarily by our commuiiity observers since organizations are better studied 
by observation, inspection of records, and interviews of organization leaders 
than by surveys. Background data on the community — descriptions of its 
demographic characteristics, major organizations, local issues, levels of 
past political activity — will help to assess the continuing effects of 
the demonstration on the communi ty-at-large , 

The EEVD effects are not limited to the school system. The EEVD may pro- 
vide incentives for structural changes within the community, and between 
non-parents and schools. Structural changes are marked by changes in who 
participates in the decisionmaking process (e,g,, peoplei who represent dif- 
ferent income, ethnic groups in the community) changing rules for the de- 
cisionmaking process (e.g., voting instead administrative fiat), and 
substantive changes in decisions (e.g., different allocations of funds). 
These types of changes may be accompanied by the development of new con- 
stituencies around new issues and shifting group loyalties. Our survey 
measurement of attitudes and propensities toward political mobilization 
in the community, careful community observation, and interviews of leaders 
should allow us to collect the data that will describe important changes 
in community structure. 
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Educational Personnel 

Educational porsonnel will be an^^lyzed in terms of their respective 
roles in the school system, such as administrators and teachers. Within 
the administrative category we include: jnembers of the school board, 
school superintendents, school principals, other administrative staff, and 
members of the EVA. Within the teaching staff we include fully certified 
teachers and paraprof essionals . 

We also intend to include state and local officials like the County 
Superintendent of Schools, members of the State Board of Education, the 
State Superintendent of Instruction, members of the State Department of 
Education, and representatives of appropriate State legislative committees 
(e.g.. Education, Finance, Ways and Means), as well as appropriate Federal 
program officers. Teachers' attitudes and responses may be compared with 
the official positions of their professional organizations — the National 
Education Association and teachers* unions. 

The social and political outcomes relevant to school personnel are also 
structural. We want to identify changing participants in the decisionmaking 
process, changing rules for the decisionmaking process, and substantive 
changes in decisions. These particular changes are primarily internal to 
the school system. While the effects may be strongest at the local school 
level, they must be traced through the system at every administrative level. 
Indeed, the impetus to change may well originate at administrative levels 
beyond the local demonstration schools. 

Relevant data will come from personal interviews and the examination 
of school records provided by the Data Management Contractor and the ad- 
ministrators themselves. Wliile we are interested in the attitudes of 
educational personnel toward the demonstration, we will weight their ac- 
tions more heavily than their statements. Administrators, especially, 
are in positions that are political as well as educational and their in- 
terview responses will be affected by that fact. Furthermore, they are 
subject to constraints which may well lead to divergences betv^een tiieir 
expressed attitudes and their actions. For instance, administrators may 
want to change the recruiting procedures for teachers but may be prevented 
by teachers* union regulations. An understanding of these constmints at 
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each status level of the educational system is best acquired by careful 
analysis of their audiences — the people to whom thay must answer for 
decisions that are made. 

Teachers and teachers* aides, on the other hand, are in somewhat 
less "political" positions than nheir administrators, because their de- 
cision space is smaller and less subject to public scrutiny. Tlieir ex- 
periences during the EEVD and their attitudes are likely to be the best 
measure of the operational impacts of the demonstration oa the school sys- 
tem. They will have the most continuing contact with changing distributions 
of students and parents, put. into effect any curricular or procedural changes 
in the classrooms, and provide administrators the basic data for records of 
classroom functions and problems. If the EEVD increases paperwork, tea- 
chers are first to be sensitive to this; if student problems increase, 
teachers should- be the first to recognize this; if parent behavior changes 
significantly, or their concerns with the school undergo change, teachers 
should be the first to know. 

We will, therefore, interview samples of teachers in each demonstration 
school, analyzing both their attitudes and experiences with respect to the 
demonstration and their movements between schools and in and out of the 
school system. Once again, these data will be compared with the data we 
acquire through thfj official channels of teacher opinion — their organi- 
zation and unions . 

Students 

The assessment of student educational achievement and growth over time 
is one of the most ii...portant issues associated with student experience in 
the demonstration schools. These data will come from the students them- 
selves (techniques of measurement and data collection are discussed in sub- 
section III-C) . Three areas of student response and experience in the course 
of EEVD are of particular interest: 

1., The actual ethnic/racial/income distribution of students in the 
schools 

2, The response of students to schools 

3, Experience with parents or significant others in the home which 
may have impacts on student educational achievement and growth. 
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Information about ethnic/racial/income distribution of students, 
will be relatively simple to acquire. The source is school records 
acquired through the Data Management Contractor. But these distribu- 
tions ax'e only the last stage in a more complex decision process, which 
begins with decisions' that parents make about where to send their childre'ii 
and decisions that schools make about admissions. Comparisons among ac- 
tual distributions after school choices are made, distributions announced 
by the EVA prior to decision time, parent and school personnel assertions 
about school integration, parent reasons for school choices, and adminis- 
trative decisions about admissions policies v/ill help us to undei'stand 
what the distributions mean and will help us to assess the impact of the dem- 
onstration on school integration. 

We have chosen not to interview jtudents directly for information on 
student response to schools and their education-related home experiences. 
Tlie target children are presumably too young to choose schools independ- 
ently from their parents. At the same time, their responses toward school 
environments (teachers, subjects, peers, rules) will prefMamably influence 
parent choices of schools. Thus during the course of the surveys, we have 
chosen to elicit parent observations about their children's responses to 
school and the substance and frequency of their school-related interactions. 

These outcomes are more educational than social/political in charac- 
ter. The home environment is of central importance in the affective and 
cognitive development relevant to school achievement. Many research find- 
ings emphasize the critical role of parents as they represent the world to 



Sep, for exauple: Benjamin Bloom, Stability and Change in Human Char - 
acteris ..ics , John Wiley, New York, 1964; Bloom, et al.. Compensatory Educa- 
tion and Cultural Deprivation , Holt, Rinehart and Winston, New York, 1965; 
R. Cloward and J. Jones, "Social Class: Educational Attitudes and Partici- 
pation," 1963, in Education in Depressed Areas , Passow, ed. ; R. H. Dave, 
"The Identification and Measurement of Environmental Process Variables that 
are Related to Educational Achievement," unpublished Ph.D. dissertation. 
University of Chicago, 1963; Robert Hess, et al., "The Cognitive Environ- 
ments of Urban Preschool Children: The Follow-Up Phase," Graduate School of 
Education, The University of Chicago; Daniel Scheinfeld, "On Develpping 
Developmental Families," paper presented at the Head Start Research Seminar 
#5, Washington, D.C., January 1969; S. L. Wolf, "The Identification and Measure- 
ment of Environmental Process Variables Ralated to Intelligence," unpublished 
Ph.D. dissertation. University of Qiicago, 1964. 
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their children, shape expectations and attitudes and provide them with 
skills. In general, those home and parent factors which have been found- 
to correlate most highly with aca^lcmic achievement as measured on stand- 
ardized tests are: 

a. Parental concern and support for achievement and learning 

b. Maternal teaching style 

c. Home resources for general learning. 

We are acquiring data primarily on the first two factors which are 
less class-biased than the tliird. However, wc will only be able to com- 
pare aggregate changes in parent-child interaction about school with ag- 
gregate changes in student achievement and growth. In order to keep sur- 
vey data confidential, we will not be able to connect individual studenL 
test scores with their parents* survey answers. Actually, it is possible 
to make these connections and still maintain confidentiality, but this is 
too expensive a task in light of the amount of evidence generated. 

CONCLUSIO N 

In the analysis of social and political outcnines of the EEVD, we are 
viewing the demonstration community and demonstration schools as open, 
interlocking systems — having both structural and functional relation- 
ships. Within the currauunity system, group effects of the demonstration 
are expected to vary by the relationship of those group?3 to the educational 
process and hence to the demonstration. Within the school system, effects 
of the demonstration are expected to vary by the relationship of groups to 
the educational decisionmaking process. As these two systems interact 
with one another around the focal point of the demonstration, responses 
pxe expeci:ed to -ary in terms of the constraints imposed by special inter- 
ests, audiences, and norma internal to each system. 

The major variables essential to understanding the structures and 
interactions of these systems are experiences , attitudes , and behavior . 
The analysis will be an analysis of process — as changes over time con- 
stitute the most critical effects to be identified. Data will thus be 
collected so that different points in time are integral to the analysis; 
historical, baseline, first year of the demonstration, second year of the 



Y-jJ^r^' demonstration, and so forth. 

hKJC 
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The substantive data that we will be collecting — organized according 
to outcome dimensions, measurement indices, data sources, data collection 
devices, data collection responsibility, and preferred forms of data analysi 
are contained in Table III-l. 
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III.B. ECONOMIC/COST OUTCOME DIMENSIONS 

PUBLIC POLICY ISSUES 

The broacj economic question underlying the EEVD is whether vouchers 
will generate broadened and improved educational options for disadvantaged 
families. This issue can be disaggregated into three general issues: 

o What will be the impact of the EEVD oin diversity and innovation 
in schools? 

o What will be the response to vouchers by public and private school 

administrators? 
o What will be the impact of the EEVD on school finances? 

These questions involve instructional, political and sociological 
considerations in addition to their economic aspects. Recognizing the 
overlaps, this section will concentrate on analysis that is directly re- 
lated to the issue of the impact the EEVD has on the supply and distribu- 
tion of educational services. 

The EEVD envisions a major reorganization of the educational market- 
place in each demonstration site. Conceivably the EEVD could lead to the 
creation of a sizable and economically viable set of private organizations 
offering educational services. This is, however, neither a primary goal 
nor a necessary condition for success of the EEVD. The relevant economic 
goal of the EEVD is an improvement in parental ("consumer'*) satisfac- 
tion. This might come about from an increase in the number of school op- 
tions, particularly options offered by profit-seeking entrepreneurs or 
non-profit community organizations. However, improved parental satisfac- 
tion might also come because public school officials perceive the need 
under a voucher system to cide.pt and provide a broader range of schooling 
alternatives and/or improved educational processes and outcomes. Thus, 
the EEVD could be successful in achieving its objectives even if there were 
no new entry nor the establishment of any viable private schools. 

In this connection it should be pointed out that in a voucher system, 
the voucher itself is not an end but merely the means to get the crucial 
element, u supply response. This supply response may be in the form of 
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entry into the market of new entrepreneurs offering preferred schooling 
options. Other responses are possible as well, however, including the 
adaption of existing public schools. The crucial issue is that parents 
have meaningful choices and schools that cannot attract parental support 
suffer for their failure to do so. 

la the following discussion "entrepreneur," "new supplier" and 
like terms are not limited to the classic businessman seeking to maximize 
his income. The entrepreneur might well be in charge of a non-profit or- 
ganization sponsored by a community group or eleeu-osynary organization. 
The motive for entry would not be profit-maximization but some form of 
public or group service. From the standpoint of economic analysis, how- 
ever, there is no formal difference between the profit-seeking firm and 
the non-profit organization in this sense. The latter has to cover at 
least some costs . Unlike the private flmi it might not demand a probable 
positive profit before it would provide educational services; the sponsor- 
ing agency might merely demand that it break even or it might be willing 
to provide a subsidy. However, if we treat profit as a variable that can 
take a positive value or be zero (break even) or negative (subsidy) , then 
we can treat all potential suppliers of educational services in the same 
fashion. The only difference is that when we ask whether a given organiza- 
tion is economically viable we have to know whether it is a private firm 
that demands some positive profit (and how much it requires) , or some other 
organization that is willing to accept a break-even or subsidy situation. 
In the latter case we have to know how much subsidy the sponsor is Killing 
to provide. 

Educational Diversity and Innovation 

The EEVD seeks two supply responses: (1) new entrepreneurs may enter 
the education sector and offer parents, particularly disadvantaged parents, 
schooling opportunities not presently available to them; (2) existing public 
and private schools may diversify their offerings in order to compete or 
to forestall entry. Under the EEVD, these new or revised educational pro- 
grams must be provided at an average cost per school no greater than the 
value of the regular and/or compensatory voucher multiplied by the number 
of students. 
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Vouchers are also intended to generate an increased propensity for 
educational innovation. We distinguish between diversification and in- 
novation. Diversity might come about by schools offering different but 
well-knovm instructional systems. Innovation implies a change in the 
state of the educational art. 

There is an organiMtional implication to more diversity and inno- 
vation. In order to be responbive to the new demands created by vouchers, 
schools may have to delegate more authority and rebpor.sib ili ty to princi- 
pals and teachers, leading to higher status and responsibilities for them. 

Critics of vouchers believe the EEVD will have adverse impacts on 
the number, diversity and innovativeness of schools. First, they challenge 
the idea that there are new and superior educational programs available 
at the price of the EEVD vouchers. They argue that the superior tech- 
nology does not exist, except, perhaps, at higher per pupil costs than 
vouchers would pay for. 

Seconds vouchers may decrease diversity rather than increase it. 
Each individual school might become more specialized and homogeneous so 
that while there was diversity among schools, within schools students 
would be limited in the choice of programs. Also, if they do not have 
to serve everyone, schools might concentrate on students from certain so- 
cial or economic groups and ignore the needs of various other types of 
students . 

The third and most significant concern is that diversity and innova- 
tions resulting from the EEVD might lower .the quality of education. Put 
differently, there is a concern that the profit motive is completely or 
partly inconsistent with socially desirable motives for supplying educa- 
tional services. There are several aspects to this concern. One is con- 
cern that vouchers would stimulate the entry into the economic sector of 
"hucksters" who would prey upon the lack of knowledge of parents about ed- 
ucation in order to perpetrate fraud and shoddy marketing practices. Or 
vouchers might lower the quality of education offered by the existing 
schools. Voucher systems could "deprof essionalize" teaching by removing 
credential requirements, destroying the present self-governance system, or 



hindering attempts to attract able professionals into the education pro- 
feosion. Other critics reason that the EEVD would create such unbearable 
administrative and planning problems for school administrators that they 
would "give up" and simply become "schools of last resort" for those chil- 
dren that private schools choose not to serve.. Finally, many worry 
diversity might "ake, the form of splitting the community into small 
sectarian, political or racial groups for schooling purposes. Schools 
would become politicized or partisan. 

Both the hopes and the fears surrounding the EEVD represent conceivable 
outcomes. The positive outcomes can be predicted from the theory of mar- 
kets assuming that: 

o Parents have substantial and accurate information about schools 
or they can obtain this information relatively easily and in- 
expensively . 

o There is a stock of unused educational technology or the state of 
the art can be substantially improved if there is a demand for 
new programs . 

o Educational costs and economies of scale are such that preferred 
or superior programs caii be delivered at costs equal to the value 
of vouchers < 

o Educational entrepreneurs will face a low degree of uncertainty 
about parental demands and the expected rate of return on their 
investments < 

However, market theory predicts adverse outcomes under the assump- 
tions that: 

o Parents lack essential information about f^chools and the quality 
of educational processes and outcomes and the required information 
can only be obtained at considerable financial or other costs, 
o No stock of unused but superior educational technology exists* 
o There are significant cost economies-of-scale in education, 
o There are strong political, religious or ideological components 
involved in educational desires of parents v 
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No priori judgment can be made about what set of assumptions Is 
more realistic, so the Impact of the EEVD on the number, diversity and in- 
novatlveness of schools cannot be predicted In advance. The Analysis and 

Survey Contractor must give close attention to these issues. Specific 

I 

questions that must receive special study are; 

o Will the educational offerings available to any specific family 

Increase or decrease in number and diversity? 
o Will the rate of educational innovation Increase or decrease? 
o Will parents have more relevant information about schools or 

will there be Increased "hucksterlsm" and "fraud"? 
o Will there be partisan indoctrination? 

o Will the EEVD alleviate or increase administrative burdens? 

o Will the EEVD increase the scope of authority and professional 
freedom of teachers or will there be a decline in teaching pro- 
fessionalism? 

o Will the EEVD lead to more or less attention to specialized edu- 
cational needs such as students with physical, mental, or home 
background handicaps on the one hand, or Intellectually gifted 
or artistic children on the other? 

Response of Suppliers of Education 

Will vouchers generate the entrepreneurial and administrative incen- 
tives envisioned by designers of the EEVD? The theory of vouchers holds 
that placing public funds for schooling in the hands of parents would 
create a demand for new and diverse types of schools that will be sufficient 
to elicit entreprenurlal response. Private entrepreneurs will set up new 
schooling alternatives. Public school officials will be more inclined to 
provide the educational processes that parents desire and 'nore concerned 
about achievement. This implies that authority and conti Lthln school 
systems will become more decentralized so that the local principal could 
serve the special needs of his nei-^hborhood. 

Proponents argue that these would be substantive changes and programs, 
not mere cosmetic or advertising efforts; opponents argue the opposite. 
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For both propositions, the new incentives are assumed to benefit poor 
and minority groups especially. The special education needs of these groups, 
it is argued, tend to be ignored due to existing economic and political in- 
centives . ' . * 

But, there are grounds to question whether these new incentives would 
be sufficient to induce the desired behavioral response. 

There are important questions about the availability of new technology and 
the costs of delivering new or superior programjs to students. Some even ; 
argue that the incentives are perverse and will lead to undesirable entry' 
into the educational marketplace of shysters and hucksters. Put differently, 
there is a concern that the goal of obtaining profits is inconsistent with 
the socially desired educational objectives. 

For that matter, the behavioral impacts on public school officials 
might be adverse rather than positive. If the public schools become 
"schools of last resort," principals and other school officials may become 
apathetic and nonresponsive because their students would have no place else 
to go. 

/ 

These issues can be studied by examining four specific questions: 

o Will private entrepreneurs in charge either of for-profit firms 

or nonprofit organizations enter the school marketplace? If so, 

what will be the nature of their offerings? 
o Will there be a decentralization of responsibility for curricula, 

educational process and educational emphasis in public school 

systems? 

o Will public school systems become more responsive to parents or 
become limited to a "captive" student body that* other schools 
do not want, and therefore unresponsive to their clients? 

o Can an economically viable private school sector exist charging 
tuition equal to the value of vouchers? What will be the motiveis 
for nonprofit organizations that enter the marketplace? Wliat will 
be the nonprofit groups' financial requirements considering pos- 
sible subsidies from sponsors or ability to operate without profit? 
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Impact on School Finance 

Some have argued that vouchers might be a means of responding to 
Serrano v. Priest and similar decisions requiring reform of school finance. 
Others have atgued that vouchers might be a legally and politically feas- 
ible method for maintaining parochial schools. But there are those who 
doubt parochial schools will subject themselves to voucher regulations. 
More generally, voucher proponents argue that the increased parental control 
over their children's education will lead to increased public willingness 
to finance schools. 

On the other side it is argued: that vouchers would increase the 
public cost of education by providing public funds for private schools, 
particularly parochial schools; that vouchers will increase administra- 
tive costs and result in losses of economies of scale and so Lead to 
higher average costs; that the result would be to divert parental inter- 
est and funds from the educational sector to other sectors, and within the 
educational sector, to divert funds away from the poor to the education of 
the affluent. 

In short, it can logically be argued that the EEVD will ameliorate 
the current school finance crisis or that it will exacerbate the crisis. 
To deal with' this topic, the ASC will have to address four basic questions: 

o Will the EEVD increase or decrease educational costs? 

o Will the EEVD increase or decrease financial support for education? 

o Will the EEVD affect parochial school financing? 

o Will funds be shifted from the education of poor students in favor 
of the education of more affluent students? 

EVALUATION METHODOLOGY 

Rand will analyze. the supply response to the EEVD by using industrial 
organization methods as the framework for analysis. Educational program 
and resource analysis at the school site level will also be employed to 
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analyze the causal determinants of the supply response behavior as mea- 
sured by changes in program offerings, changes in resource allocation 
patterns and changes in school finances. 

The essence of the industrial organization analytic approach is that 
marketplace phenomena are divided into three categories: structure, be- 
havior and performance variables. The causal chain, following economic 
theory, is chat the structural conditionti of the market determine entre- 
preneurial behavior which in turn determines how well the market performs 
as measured by various indices. The relationships among variables in the 
three classes, however, can take various forms. 

To depart from the specifics of education and look at markets gener- 
ally, a market structure characterized by a large number of suppliers each 
with a small fraction of total sales will tend to lead to competitive be- 
havior or entrepreneurial conduct and the performance of the market will 
be characterized by low prices, non-excessive profits, low selling costs 
and a desirable rate of capital investment. Conversely, markets with small 
numbers of sellers typically evidence explicit entrepreneurial collusion 
or implicit agreements to "live-and-let-live" or "follow-the-leader." 
These behavioral policies in turn frequently lead to socially undesirable 
prices, profits » selling coats and inv^^^stment . Industrial organization 
theory provides paradigms of different structural, behavioral, and perfor- 
mance combinations. These models are helpful in organizing research, but 
there are so many alternative relationships possible among the various 
aspects of a market that careful empirical study and thoughtful analysis 
is required to delineate the actual causal relationships that apply in any 
specific market. For the EEVD, where social and political motivations will 
likely play important roles in addition to the desire for financial gain, 
it will be especially important to analyze causal relationships among struc- 
tural^ i3*ihavioral and performance variables. 

Market structure refers to the organizational characteristics of the 
market. Structure usually is affected only marginally by the participants 
in the market and is less volatile than behavior or performance. Market 
structure characteristics generally determine actions of entrepreneurs 
rather than vice versa. The EEVB can be conceived of as a significant » 
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discrete change in one important structural condition, the demand function. 

The structural variables that are important for the analysis of the 
EEVD are: 

o Nature of the demand for educational services 

o Number of schools available to different categories of parents 
o Differences in the educational offerings of schools ("product dif- 
ferentiation") 

— locational differences 

— curriculum differences 

— educational process differences 

— "public," "private" and "parochial" characteristics 

— political, religious or social differences 

o The "barriers to entry" of new schools or to school change 

— degree of autonomy of local public schools to respond to changes 
in market demands 

— start-up and capital costs 

operating costs — economies of scale and economies of plant 
utilization 

— legal requirements for entry or change 

Market behavior or market conduct refers to the policies and procedures 
used by buyers or sellers to adapt or adjust to the market hanges. In the 
EEVD context the buyers will be patents and the ASC will need to know: 

o Wliat parents know about schools 

o Wliat parents (buyers) want to know for decisionmaking purposes 
o What criteria parents use to select schools 

o How much control parents want over decisions about their children's 
schooling. 

On the supplier side — the private and public schools in the relevant 
areas — the analysis will cover: 

o Inves tmen t pol Ic les 

o Tuition and other price policies 

o Admission policies 
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o Policies with respect to determining curriculum, educational pro- 
cesses, emphasis and other elements of the "product*' offered. 

The analysis will also have to be concerned with how these policies 
were determined. Do schools simply use tradition-based policies? Is 
there collusion among educators? Is there an independent, rivalistic 
posture? Who makes policies? How centralized or decentralized are the 
decisions? How do the rules of the EEVD actually affect tuition and ad- 
mission policies? 

Performance variables measure how. efficiently a market is meeting 
consumer demands. The pertinent variables are: 

o Prices, cost margins and profit rates on investment 
o Promotion expenditures 

o Rates of innovation and curriculum change 
o Nature and range of the products offered 

o Nature and range of perceived student outcomes (See subsections III.C, below) 
o Entry into and exit from the market. 

None of the variables — including the structural variables — are 
likely to remain completely fixed over the duration of the demonstration. 
The pattern of consumer demands for education is likely to shift, rules of 
behavior may well shift from traditional rules of thumb -to collusion or 
rivalistic conduct, profit patterns may change and there will be a changing 
time pattern of values for most of the other variables. The greatest ana- 
lytical interest will be in the changes in the structural variables of the 
market - 

Structur al Variables 

— ' ' — ! 

Demand Patterns , The structure of parent (consumer) demands for edu- 
cational services specifies a set of constraints on any school official's 
behavior. Private profit-seeking entrepreneurs cannot be expected to offer 
schooling opportunities that parents do not desire nor schooling opportuni- 
ties that cost more than parents are willing or able to pay. The same prin- 
ciple applies to public schools required to support themselves with voucher 
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revenues. Non-profit, sponsored schools might be able to run at a loss, 
but even here there is some subsidy that will exceed the interest or 
financial resources of the sponsor. Thus the demand function is, under 
a voucher system, a basic constraint on profit-seeking, non-profit and 
public schools alike. 

The pattern of demand also provides a normative standard against 
which to measure market performance. If entrepreneurs are not providing 
the type of products people want, the market has failed. 

As noted previously, demands may well change during the EEVD as parents 
gain more experience in making choices among alternative schools". Any such 
changes and the interactions between demand changes and supply responses 
are very significant subjects for analysis. 

The basic technique for measuring parental demands and changes 
therein is the parent/community surveys described above, supplemented 
with observational data such as the schooling preferences revealed when 
parents actually make choices. 

Actual choices may differ from stated preferences and, in that case, 
will be the subject of further observation and analysis. 

Number of schools available * A basic set of data is the number of 
schools available, the number of students they can each accommodate and 
the extent to which they are operationally and personally acceptable 
schooling options for different classes of students. An increase in the 
number of true alternative schooling options would be an important favor- 
able outcome of the EEVD. A school in a given locality may not be a mean- 
ingful option to a parent because of its location or some implicit require- 
ment or "image." Survey data plus some informal observations and interviews 
will likely be required to determine the correspondence between the set of 
all schools and the set of feasible options for each identified group of 
parents . 

Differences among product offerings . Schools do not offer identical 
curricula and services. The EEVD is an attempt to increase diversity. 
Ttiere are two economic aspects to these differences in school "product." 
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On the one hand, diversity among schools may reflect a response to dispari- 
ties among parental demands and student needs. On the other hand, a firm 
can insulate itself from competitive forces by convincing customers that 
its products are not like those of other firms. 

The analytic task is to determine: (1) what are the differences 
among the schools? (2) do these differences reflect underlying differ- 
ences in advertising "images,'^ (3) do the differences reflect responses to 
disparate parental demands or attempts to limit competition through market 
segmentation? The first subtask is relatively easy and can be answered 
by developing educational offerings profiles through observation and 
interviews at different schools. The last two subtasks are progressively 
more difficult, requiring examination of "advertising" techniques, observa- 
tion of the technology used and consideration of parental answers to the 
surveys . 

Barriers to entry . The supply response will depend upon restrictions 
to entry in the educational marketplace, or, as it is sometimes called, 
the heights of the barriers to entry. For the adapting public school the 
major barrier will be obtaining the authority needed to respond to market 
isignals. Few principals have this authority now and can help establish 
whether changes in this barrier take place. There are likely to be three 
significant obstacles to private entrepreneurs: 

o Legal requirements such as teacher certifications, curriculum 

requirements and the like may restrict supply response, 
o Entrepreneurs must expect to amortize start-up costs and must be 

able to obtain access to capital markets; the higher these costs 

the-greater the barriers to entry, 
o In order to enter the market effectively, the economies of scale 

and of plant utilization .must be such that the new schools can 

operate profitably at the voucher price. 

Data on these barriers can be obtained by interview and by cost infor- 
mation generated by the Program and Resource Analysis. 

The individual school is the focal point for the Program and Resource 
Analysis. Aggtegative data for the Individual schools will provide the 
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basis for assessing the impact of the EEVD on the flow of dollars 
to education within the boundary of the demonstration. 

The Resource Analysis Plan is designed to provide (1) a description of 
the baseline resource allocation by school within each demonstration area, 
(2) estimates of the resource impact and the dollar cost of changes in the 
educational programs for several levels — subject, grade, school and dis- 
trict and (3) a picture of changes in the dollar flows, both revenues and 
expenditures, within the district < 

The Resource Analysis Plan views schools as organizational entities 
that use resources in different mixes to produce instructional and other 
educative programs < These programs are regarded as an intermediate product 
with student performance as a final outcome. The approach can be illus- 
trated as follows: 



Resources 



Combine to Yield 



Programs 



Students 
Staff 

Facilities 

Equipment. 

Materials 

Supplies 

Services 



Instructional Strategy 

(Conventional classroom) 
(Open classroom) 
(Individualized instruction) 
(Team teaching) 
(Other strategies) 



Curricular 

(Subject) 

(Lesson plan) 
Non-cur ricular 

(School activities) 

(Extr^-school activities) 



The instructional strategy is the process by which resources are con- 
verted into program outputs. Particular attention will be paid to defining 
processes and to determining the extent to which the identified process for 
any program is adhered to in conducting the program of instruction. 

For each school within the demonstration boundary, the programmatic 
output will be detenained. For grades 1 through 6, the estimated time al- 
located to each "program" will be determined in the manner shown in Fig, III-l. 



The methodology to be used is described in S- A. Haggart, Program Cost 
Analysis in Educational Planning ^ The Rand Corporation., P-4744, December 1971 
and also in Appendix C of this report* A planning cost model for educational 
programs that will be useful in the early stages of organizing the resource 
information for the analysis of the supply response is also described. 
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For grades 7 and 8, where there is a matching of the programs with the more 
discrete subject-related periods, the schedule of classes will be used. 
The resulting program description of the output of each school will be ana- 
lyzed in terms of its resource requirements and dollar cost. The program 
and resource analysis will be used. The resulting program description of 
the output of each school will be analyzed in terms of its resource require- 
ments and dollar cost. The program and resource analysis will be conducted 
each year of the EEVD so that changes can be tracked. 

In assessing the fiscal impact of the EEVD within the demonstration 
boundary, a model developed for the California State Department of Educa- 
tion will be used to "crosswalk" the budget as presented in traditional 
format to a program budget. An example is shown in Fig. III-2; the numbers 
across the top represent the traditional budget category classes. The 
categories for a program budget are listed down the left side of the table. 
The model also permits the estimation of the cost of all educational pro- 
grams offered by the suppliers of education. The major inputs are: number 
of students, material and equipment costs; salary and wage schedule for 
staff i class size^ (all of the proceedings are by subject and grade); para- 
professional hours per class-hour; student attrition; and teacher-equivalent 
hours per week. 

The basic data required are: 

o Schools — Number, size, space, usage 

o Students — Enrollment, by grade, class, socioeconomic status, 
mobility 

o Teachers — Number, salary, training, turnover, transfers 
Class load, extra-curricular activities 

o Equipment and materials — Available resources, usage, 
program requi reTnputs 

o Programs — ; Acuivity output 

o Revenues — Revenues and sources 

o Expenditures — Budget and expenditures by school 

o School factors — Administrators per staff member., clerks 
per admirji St rater , etc . 
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Market Behavior Variables 

P arental behavior . The key issues here are the decision rules used 
by parents and parents' informational base and schooling desires. Such 
information will be obtained by the surveys and data collected by the 
ISC and EVA. 

P rivate profit-seeking and nonprofit entrepreneurs . The key issues 
are how suppliers determine investment, price and product decisions and 
their admission or "marketing" policies. The differences among the 
motivations and incentives for profit-seeking, schools, non-profit 
organizations and units of the established public school system will be 
investigated. Required data will be obtained by interviews. 

Public schoo l officials. The key issues are how public school 
officials determine, the products they offer, how decisions are made, 
and which officials actually make them. Such data will have to be 
obtained by interviews. 

Performance Variables 

Parental (consumer) satips faction . The ultimate test of the 
performance of any market is its ability to satisfy consumer demands. 
This criterion is particularly important for educational marketii in 
general and the EEVD in particular. To reemphasize a point discussed 
earlier, the success of the demonstration does not depend necessarily 
upon the success of new schools but rather, in part, on the ability of 
the demoitstration to increase the satisfaction of parents with the 
education thetr children receive however this is accomplished. 

In most industrial organization studies, change in consumer 
satisfaction is measured indirectly. It is inferred from other variables 
such as shifts in sales from one firm to another or changes in costs, 
prices or profits. In the markets involved in the EEVD consumer 
satisfaction will be examined directly by means of periodic surveys 
of parents. 
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It is conceivable that in any site-coininunity the established 
schools might react to new competition or react to forestall the 
entry of new competitors by becoming more responsive to parental 
demands, adapting programs or in other ways attempting to increase 
parental pleasure with the existing schools. Thus, it is possible that 
market performance as measured by the expressed attitudes of parents 
could increase without any change in the number or ownership/sponsor- 
ship of schools in the community. 

Price-cost margins and return on investment . As just discussed, 
market performance could improve without any change in the number or 
type of schools in the comm?jinity. The important requirement is that 
competition or the threat of competition be a credible force in the 
minds of school officials. The ability of parents to shift their 
Children among the existing schools may be sufficient to establish 
this credibility. Competition, however, may require or may be 

improved by the existence of a group of new profit-seeking or nonprofit 
schools. If such a group of schools is established it will be vital to 
examine their economic viability and competitive potential. The key 
variables here are the price-cost margin and the rate of return earned 
on investment. Competition, if it is effective and viable, will lead 
to low price-cost margins but also to competitive rates of return on 
investments. If the priee-cost margin is negative or very low or the 
return on investment is less than that that can be earned in other 
fields, firms can be expected to leave the field. If the price-cost 
margin is high or there is a supra-normal rate of return on investment, 
entry of new supplies will occur. Thus, the price-cosit margin and the 
recurn on investment are tests of how well competition is working and 
also a basis for forecasting future entry and exit of schools. 

S ' 

As noted above^ for sponsored nonprofit firms these could be 
negative and the school might still be a viable and effective 
competitor. 
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The return on the investment of investors in private schools has a 
special importance for the analysis of the effect of tlie EEVD on the educa- 
tional marketplace. A major question about the demonstration is whether an 
economically viable set of schooling alternatives can be developed 
within the financial constraints of the EEVD and EVA. Some observers 
doubt that the value: of the vouchers established by the EVA will be 
sufficient to support an educational sector with a variety of schooling 
alternatives. Therefore, it is vital to be able to answer the question 
of whether the schools that operate during the EEVD appear in sufficient 
economic health that they can be expected to continue r 

A profit-seeking firm will require a positive margin between 
tuition and costs if it Is to survive, A nonprofit firm may, if its 
sponsors will provide a subsidy, survive although it runs a deficit. 
Under the EEVD rules a school in the existing public school system 
must reiy on voucher receipts to cover its costs but it does not require 
R positive profit to be viable. Thus, it is possible that there could 
be a competitive set of public schools that seek merely to cover costs > 
a S£^t of sponsored nonprofit schools that seek merely to cover non- 
subsidi:,:ed costs and a set of profit-seeking firms that seek revenues 
graa^zer than costs. If all three groups can achieve their goals under 
the voucher system, there will be a highly competitive situation. If 
prioe-cost margins will not be attractive to profit-seeking firms, then 
the issue of required or likely subsidies for non-profit schoolA becomes 
an important consideration. Also important in this context is the 
extent and efficacy of competition among units of the public school 
system. 

If the profit-seeking schools turn out to be economically prosperous, 
the Analysis and Survey Contractor should then investigate whether the 
profits earned by these schools are excessive. It is conceivable that the 
EEVD could provide a windfall for private schools. EVcn if the nominal 
profits are not excessive, the ASC should go further and look at salaries 
and expenses. The norainal profits shown on formal balance sheets may be 
low or negative but the owners may be taking the profits in the form of 
excessive salaries or perquisites. 
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For the basic analysis of rates of return on investment, profit', 
and the viability of schools Rand will rely on data generated by the 
Program and Resource Analysi^s described above. This analysis will be 
reenforced, checked and supplemented by interviews with officials of t\\e 
schools in the district. Tn addition Rand will also use the data 
collected b., the EVA in its financial analysis of the schools. 

Promotion cos ts . Informing the public of one's offerings ic a 
necessary part of conducting a school, whether the school is profit-seek- 
ing, nonprofit, or public. The cost of informing the public of 

one's product iu a necessary business expense but exceptionally high 
sales promotion costs indicate that competitive rivalry among firms is 
taking the form of advertising wars rather than price reductions or 
product improvements. The prior discussion of data sources applies here. 

Rates of innovation and curriculum change s. Competition is supposed 
to stimulate responsiveness and change and this is a major goal of the 
EEVB. Measuring innovation and change is never easy. Significant changes 
or innovations have to be distinguished from trivial or cosmetic changes. 
Moreover, the numerical scale and weighting system for innovation and 
process change is not obvious. The ASC must make sufficient classroom 
and other observations to have an understanding of the evidence generated 
by objective measures of proc<^ss change. 

Rand proposes a straightforward approach to the measurement problem. 
The existing set of processes will be specified prior to the distribution 
of vouchers. Then a tabulation of the number and type of educational 
process and policy changes and the number of actual and potential students 
affected will be maintained. 

Nature of the product offered . A basic issue is whether schools are 
providing the services desired or whether by collusion, tradition or 
nonawareness students have Hobson's Choice and have to take what schools 
offer. The basic te hnlque used will be comparison of parent survey data 
and the data on schooling options. 

Entry into and exit from thr- market . The EEVD assumes that the 
"invisible hand" will work to attract entrepreneurs of either profit-seeking 
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fjrins or non-prcflt organizations and public officials who can offer 
schooling options preferred by the public and that those offering less 
preferred options will have to change their products or leave the 
educational sector. Entry into and exit from the market, and the 
reasons for such decisions, will be monitored through personal interviews. 

The significance of a new school entrepreneur entering the ruiarket 
is obvious. The significance of a school entrepreneur leaving the 
market because of lack of financial support is harder to Interpret. 
For purposes of the EPIVl), exit due to financial difficulty is not 
necessarily a negative outcome. One of the basic ideas behind the 
HEVD is that schools should be accountable to parents and the wider 
community and schools that cannot attract the f Inanclal support of 
parents should leave the educational sector* Exit, therefore, may 
indicate that the EEVI) was successful in achieving its goals. 

Exit, however, may indicate that schools cannot operate charging 
a tuition equal to the value of vouchers. This Interpretation would 
call into question the design of the demonstration. 

Exit, moreover, may reflect neither lack of parental support nor 
an inadequate relationship between necessary costs and the value jf 
vouchers. Exit may merely reflect some factor Irrelevant to the EEVD 
such as illness of the headmaster. 

Considering that the EEVD will last only five to Stiven years and 
that new schools may not be Immediately established, if they are 
established at all, there may be no exits to o^hserve. Lack of exit for 
reasons discussed previously is not a measure of EEVD failure. But If 
exit should occur it will be important to interview the school personnel 
involved as well as parents and analyze the program and financial 
conditions of the school* Only with such data can the necessary 
interpretations be obtained. 

Timing of Changes 

The economic theory of markets that underlies the design of the 
EEVD is essentially timeless ; all changes and responses are assumed 



to take place instantaneously* In real markets, however, changes,, 
adaptions and responses have a time dimension • The information 
available for decisions is not immediately available but must be 
obtained; uncertainties attend every decision and decisionmakers may 
wait until time clarifies the situation. Therefore, for every change 
in the structure of the market, or in the behavior of parents of 
school officials (profit-seeking, nonprofit, or public) it is important 
to know; 

o Why did the change occur when it did rather than earlier 
or later? 

o What were the antecedents of the change? Would these 

always have to precede a change? 
o What information requirements were involved in the 

change? 

o What were the uncertainties involved in the decisions 
that brought about the changes? 

The data to answer these questions will vary with the change or 
decision to be investigated and so cannot be specified at this time. 
Important data sources, however, will be the surveys and the interviews 
with those participating in the demonstration. 

Adverse Outcomes 

The question of whether the EEVD achieves its positive objectives is 
most important. But it is also important to investigate unintended 
negative impacts. Of the various possible adverse outcomes discussed above 
it seems worthwhile, to accord ^special attention to two of them. The first 
is "hucksterism*^ or the possibility of fraud and exploitation of parental 
ignorance. Rand proposes to check carefully on this possibility through 
its parental surveys and through its analyses of curricula and 
educational programs. The second adverse! outcome is the possibility that 

Alternatively, one can ask whether economic incentives are compatable 
with school quality and educational needs and objectives. 
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product diversity will take the form ot l(Aeological » sectarian-* racial 
or social-class specialization. An increase in social fragmentation 
or a decrease in educational concern for handicapped children or 
students with other special needs would be a disturbing consequence 
and require*? careful invest igat ion » 

Note that financial failure of schools is not listed as an adverse 
outcome. As noted above, exit of schools in some cases mlghi: be an index 
of EEVD success. A large number of exits by schools with popular 
programs might, however, indicate an EEVD design problem and this 
possibility must be evaluated. 

* 

D ata Requirement s an d Analytical Techniques 

The basic EEVD outcomes to be investigated, the data require- 
ments and tlie preferred analytical techniques are shown in Table 
111-2, This table and the preceding discussion assumes (1) a program 
of non-intrusive Rand interviews in each participating site; (2) an ISC 
system capable of providing detailed and reliable dala about educational 
programs and costs. If it is not possible to obtain Interview data and 
Rand has to rely on data generated ap adjuncts to other parts of the 
program, the economic analyses cannot be as thorough or quantitative in 
nature. The resulting reports would generally be limited to descriptions 
of structural changes with little analysis of the reaa^ins for these 
changes and what they might imply for other applications of the voucher 
systemr 
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III,C. EDUCATIONAJ. OUTCOME DIMENSION S 

This subsection discusses the measurement of educational outcomes, 
principally in the cognitive and affective domains. The ASC will 
also evaluate the student 's educational envi ronTOnt through class- 
room observation designed (1) to assess the congruence between 
programs as described and programs as implemented and (2) to yield 
a measure t*f teacher effect on student growth. 

We assume that the evaluation should not intervene in the 
demonstration. Therefore, evaluation will not be used as feedback 
for program Improvement. We also assume that the evaluation, like 
tlie demonstration, will be an evolutionary process. The evaluation 
framework will probably not be altered during the course of tiie 
demonstration, but the instruments (especially in the affective 
domain) are highly experimental, so more tlian one instrument will be 
used to test each variable* After data have been analyzed, decisions 
will be made about subsequent data collection strategies. Instru- 
ments yielding useful information will be continued; those judged 
inadequate will be discontinued, and others substituted. 

Achievement tests will be administered to all students; other 
measures of student growth will be used on a small sample the first 
year. All students will be tested on some measures, but sub-samples 
for intensive study will also be selected, with each sub-sample ad- 
ministered a different but overlapping set of tests. (The sampling 
scheme is detailed below.) Those instruments showing the most promise 
will be used to build a test battery for further use on a large sample 
in future years. The unit for intensive study will be the classroom. 

Because of the long-term nature of the voucher demonstration, stu- 
dents in the early grades will by definition be followed throughout the 
term of the demonstration. However, a sizable number of stuci'entB, par- 
tic-jlarly those in the upper grades, will participate for shorter 
periods of time. The potent^ "^1 impact of lasting effects will be lost 
unless th^se students can be followed beyond the eighth grade. The 
necessity for assessing the impact of early educational changes on later 
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achievement and attitudes poses minor problems in a unified school 
district and major problems in an elementary district. 

It can be anticipated that a unified district will be interested 
in finding out whether changes in the elementary ochools affect its 
secondary schools. It should therefore be relatively easy to arrange 
to follow voucher demonstration students as they progress beyond the 
eighth grade. A more §erious problem arises in the case of the demon- 
stration site, such as Adam Rock, which is an eler^ntary district and 
whose students go to another district after the eighth grade. 

Early in the implementation phase, it will be necessary to make 
arrangements with the site districts to follow tEVD students when they 
leave the demonstration. 

The first part of Sectinn III.C specifies baseline data to be col- 
lected. The second part discusses cognitive measures; the next is 
devoted to affective measures; and the final part specifies the design 
for classroom observation. Table at the end of Section III.C pre- 

sents the data specifications and Management relevant to the assessment 
of educational ouEcom'is. 

BASEL I NE DATA 

In order to assess the impact of changes that occur during the 
demonstration, a thorough documentation of key educational trends must 
be made. The following information will be collected school by school 
for the rurrent year, and for as many of the four previous years as 
available . 

o Distribution of students by race, SES, ability 
o Student achievement levels » by race, SES 

o Number , character , goals , and target groups for educational 
innovations and alternatives introduced 

o Key administrative practices (admissions, discipline, measure- 
ment and accountability, e'tc), and the reasons for any impor- 
tant changes 

o Important (e.g., district-wide) changes In curriculum; the 
reasons for these changes 

o Non-curriculuir school program activities, and Important changes 
in the kind or level of these activl,*:ies 

ERIC 
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o Won-academic school services (e.g,, health care, nutrition, 
use of school plant by community, etc.) 

o Stafi characteristics and any changes therein: teacher train- 
ing, recruitment and credentials requirements, salary levels, 
use of paraprofessionals, ratio of guidance: and administrative 
staff to classroom teachers, staff turnover , age distribution. 

o Teaching techniques: team teaching, subject specialists, 
individualized or small group instruction; use of supple- 
mentary and AV materials. 

o Ratio of adults to students in classroom 



C OGNITIVE MEASURES 

Valid objections have been raised to measuring the success of educa 
tional programs solely on their contribution to reading and arithmetic 
achievement. Creativity and writing skill, for example, are recognized 
as important, but because standards are difficult to define and success 
criteria difficult to establish, they are often neglected. 

This section deals with the measurement of achievement in reading 
and aritlimetic (basic skills). Alternatives to pre- rnd post-testing 
are discussed, as are the problems associated with the use of gain - 
scores. Despite the statistical problems, the use of pre- and post- 
tests to measure achievement is recommended. We tlien specify a pro- 
cedure for measuring student progress in cognitive skills representing 
a higher level of understanding and ability than that measured by 
standardized tests . 

The generally accepted procedure for measuring program succejis is 
to administer an achievement test in the fall (pre-test) and again in 
the spring (post-test). The difference between the two scores is 
regarded as an indicator of academic progress attributable to the 
program. On its face, this is a logical argument; yet, for some time 
now , leading psychometrlcians have expressed serious reservations about 
this kind» of gain score, and any plan for measuring achievement must 
therefore veigh the possible benefits of alternatives and additions to 
normative tests. 
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ALTERNATIVES TO NORMATIVE TESTS 



Criterion-Referenced Tests 

Normative and criterion testing each have an important place in 
evaluation — normative testing provides information for summatlve eval- 
uation; crtterion testing Is used in formative evaluation. As diagnostic 
tools used to plan the next instructional sequence, criterion tests nrc 
excellent. No student has the opportunity to let work slide until it is 
too late for corrective action, and frequent testing provides the teacher 
with valuable feedback. 

However, criterion tests are curriculum specific; no commercial tests 
are currently available. Nor should the evaluator undertake to write 
the items, for if objectives or curriculums change, new items have to bo 
written constantly to reflect those changes. This is properly the task 
of the educational community, not of an outside agency. 



A P r e d i c t i ve - Ve r i f i c a t i on Plan 

Theoretically, a sound approach to achitivement testing would be to 
administer an individual intelligence test (such as the Binet or the 
SISC) as a pre-test, and a standardized achievement test (such as the 
California Test of Basic Skills — CTBS) as a post-test. In this way, one 
could derive an expect.^-d score for each student: based on a measure of 
general ability. A student *s progress would be measured against his own 
ability, not against that of a normative group. For those meeting or 
exceeding the expected score, a program would be judged successful. For 
Chose who did not achieve up to expected levels, additional information 
would be sought in an effort to explain what happened. Steps would be 
taken to help the student do better, as feasible. 

Unfortunately, this type of testing is expensive; moreover, intelli- 
gence testing is highly suspect in many communities with large minority 
populations (it is not obvious that admiuistering a test in Spanish would 
overcome objections to cultural bias) . 

^ An alternative procedure might be to Ubc group intelligence test, 

ER^C California Short-Form Test of Mental Maturity (CTMM) • An "Anticipated 
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Achievement Grade Equivalent" score to predict a student's performance 
on the CTBS is derived from his score on the CTMM, Age, grade and sex 
are taken into account. Using the CTMM in the fall a.id the CTBS in the 
spring a student's achievement can be compared to a nationwide sample of 
students who have similar characteristics. Gross discrepancies between 
an individual's anticipated and actual achievement are easily spotted. 
However, this procedure would not overcome objections that might be 
raised to the administration of individual intelligence tests, and is 
beset as well with numerous practical and theoretical problems chat 
argue against the probability of achieving reliable results. 

In view of the problems raised by these alternatives, and the small 
likelihood of arriving at solutions acceptable to teachers, administra^ 
tors and parents, it seems best to reject them in favor of improving 
well-known ;jnd generally accepted pre-post testing, so as to minimize its 
objectionable features , 

PRE- AND POS T-TESTING 01' ACHIEVEMENT 

The results of any achievement test must be analyzed to obtain the 
truest possible assessment of student progress. Four basic concepts 
must be borne in mind when considering the use of gain scores, especially 
if a large proportion of the students ore educationally disadvantaged, 

o The scores of large numbers of low-scoring students will be so 
low as to be indistinguishable from chance scores, and for th^m 
there is, therefore, no real beginning score. 

o All scores will be affected by a regression to the mean^ so that 
on the post-test many Tow pre-test scores will be raised by 
chancB, and aiany high scores will show a loss, 

o The use of a raw gain score for individuals dees not take account 
of the error term associated with each of his test scores, and 
the intercorrelation of pre- and post-test errors, 

o There is no necessary causal relationship between student gain 
^nd instructional strategy — additional information about the 
student s p/^ist and current educational experiences is needed 
before that relationship can be established, 

A number of authors have questlo.ied the validity of raw gain scores 
and it has been shown that they vce extremelj biased esv;imates of true 
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gain. The basic problem arises because of errors of measurement 
associated with both pie- and post»-scores , and correlations between 
them. Cronbach and Furby offer the most sophisticated method for 
estimating true score. 

School Status 

In addition to information about students, a continuing measure is 
needed of each school's relative academic standing. Annual (spring) 
testing will hfi sufficient for this purpose. Of relevance here is our 
ability to observe the ranking of a school in relation to its popularity 
among parents, as defined by applications for admission and requests for 
transfer-out . 

Choice of Achievement Tests 

Because we wish to describe demonstration outcomes as precisely 
as possible it is more important to maintain comparable historical data 
than to focus on any particular achievement test because of its techni- 
cal merits, which are likely to be marginal in any event. Tests currently 
being used to measure reading and arithmetic achievement should therefore 
be used during the voucher demonstration. If they are not routinely 
administered in every grade, provisions should b€= riiade to complete the 
battery. 

It is particularly important that the appropriate level of test be 
administered. If historical data indicate, for example, that students 
in the fourth grade ri qd on the average at second grade level, then a 
test appropriate to the second grade should be used. In this way, the 
number of scores achieved by chance is raducfed, and a mor^ accurate 
picture of what \ e students know is obtained. An incidental benefit 



For example, see Harris, C. W. (ed.), Problems in Measuring Change , 
University of Wisconsin Press, Madison, Wisconsin, 1963. 

Essentially their technique is to use multiple regression to remove 
the effect of the pre-*test score and its error on the post-test score, and 
vice versa. The technique also takes into account concurrent scores on 
othev tests to improve the estir^ate. Cronbach,. L. J., and L. Furby, "How 
Should We Measure ^Change' - Ol Should We?," ?fiyc hology Bulleti n, 74, 1970, 




pp. 68-80. 
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is that fewer students are subjected to a failure situation, with all 
its attendant unpleasant effects. 

Test Instructions an d Procedures 

While we are aware chat test procedures in school districts rarely 
conform to good practice, in keeping with thr assumption that the evalua- 
tion is not tc be an intervention^ we do not make any recommendations about 
testing procedures. Rather we intend to monitor district implementation 
of the achievement testing in order to observe whether the demonstration 
itself leads to improved procedures. 

We shall cake cognizance of the following factors as evi^Jonce of 
change: 

a. Increased use of the appropriate level of the test, rather 
than the use of grade level tests for populations well below 
standards of performance. 

b. Better preparation of disadvantaged s*-udents for test-taking 
as eividenced by familiarizing them wit:- test formats aud 
instructions . 

c. Familiarization of test administrators with the instructions, 
including standardization of responses t;^ questions about 
guessing. 

d. Better adherence to time allowances specified in test manuals. 

e. Provision of pood ^laysical surroundings for test-taking, include 
ing ventilation, lighting and spacing of students to raiuimize 
chances for deliberate o>: iPuuv^^rtent copying. 

HIGHER ORDTjR LEARNING 

Educational evaluations rerely measure higher level cognitive learning 
(creativity, abstract reasoning, problem solving, etc.). This is partly 
dir to the lack of good standardized group tests of most higher cognitive 
ibllities, and in part to the emphasi^j on basic reading, arithmetic and 
specific content material that has dominated edu^.ation objectives in recent 
years. The EEVD has potentially important implication in tnis context for 
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two quite separate reasons. First, some parents may place high value on 
higher order learning, and schools may respond by emphasizing these objec- 
tives more in curricula design. Second, whatever the views of parents may 
be, it is an important aspect of student achievement. 

Abstract Reasoning 

Although chere are some standardized tests for abstract reasoning 
and problem solving in mathematics and science courses, these are tied 
to specific course content and are largely designed for use in liigher 
grades. Reasoning ability is sometimes measured by tests of general 
intelligence or mental ability. Of these tests, the UCLA Center for the 
Study of Evaluation rates the Otis-Lennon mental maturity test higher 
than any of the other tests reviewed; this test will be given to a sample 
of classrooms. 

Another test for reasoning ability is the Primary Mental Abilities 
test, which reports somewhat different sub-scores than does the Otis- 
Lennon. Although this test is not rated as high as the Otis-Lennon, we 
feel that it should be given to a small sample of students who are not 
^iven the Otis-Lennon. These tests are not being used as intelligence 
tests; our .interest is in the profile of sub-scores. 

Creativity 

Although creativity is difficult to define in operational terms, 
there is a large body of research on the topic, and much is known about 
the characteristics and needs of the creative person. Creativity is not 
necessarily associated with high Intelligence, and it appears that the 
learning environment of the creative individual must he different in many 
respects from that of his lees creative countti mparts. 

Unf or tunate?,y, research on creativity has not resulted in any simple 
and highly reliable measures. However, some standardized tests for 
creativity are available, and although their validity is not as high as 
one would like, they appear to be worth pursuing in the EEVD. One such 
test that is rated higher than others by the UCLA Center for Evaluation is 
the Torrance Test of Creative Thinking. This test reports sub-scores 
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on nine categories of creative thinking > and is promising enougli to 
administer on a sample basis. 

Sca^ljj ig of Student Performance 

Essay examinations and written reports are useful in measuring a 
broad range of student acliievement , especially achievement that is highly 
relevant to future academic success. Unfortunately, the grading of tliis 
type of material is extremely unreliable. i'art of the problem stems from 
the multi-dimensional characteristic of complex learning activities o 
Teachers usually report a single grade wliicli is a subjectively weighted 
sum of achievement in several dimensions (e.g., in writing: vocabulary, 
clarity, neatness, spelling, originality, etc,)- 

In spite of the difficulties inherent in scoring essay type material, 
some kinds of cognitive activity simply cannot be measured adequately in 
other ways. Standardized tests (and al.l short answer tests) at best 
measure only retention of specific content material. We will approach 
this probl«^,m by continuing to develop a method proposed by Rand for 
measuring performance across a wide range of student activities such as 
writing, problem solving, and artistic expression. Tl\e method is based 
ca a judge's evaluation of student achievement levels as reflected in 
samples of their vork collected periodically during the EEVD, sc that 
the scaled scores can be used to indicate both status and change. Because 
iihis eifort is in the development stage*, it will be used on a relatively 
small sample during the first year of the demonstration. The scaling 
method is detailed in Appendix D* 

AFFECTIVE MEASURES 

Interest in the improvement and measurement of affective states 
(motivation, attitudes, self-esteem, self-^iwareness , ha|jpiness and other 
personality variables) has steadily increased in recent vears, motivated 
in part '•y Lhe lack of success in modifying cognitive achievement through 
'^standard" educational innovations. The importance of affective growth is 

Donaldson, T. S., Subjective Scaling of Student Performance , The 
Rand Corporation, P-4596, 1971. 
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defended on the basis of two arguments. One view contends that affec- 
tive factors ire i-.portant because they are beiieved to be the major 
determinent of cognitive achievement, and there is considerable experi- 
mental evidence supporting this view. The other view holds that growth 
in affective rather than cognitive faccors is the more relevant goal of 
education. These views are certainly not mutually exclusive, and most 
educators agree that noncognitive factors are importa.^t for both reasons 
In fact, the distinction between affective and cognitive achievement is 
rather artificial: attitudes and motivation have ^rong intrinsic cog- 
nitive components, and cognitive skills have strong intrinsic affective 
components . 

Despite this 'owing discourse among educators about the importance 
of affec ive growth, the successful implementation and measurement of 
affective objectives in the schools remains disappointing. Affective 
objectives must be stated in the development of curricula, then trans- 
lated into classroom activities. This process is a difficult one; 
authorities do not agree on definitions of affect nor do they agree on 
the relative importance of affective objectives. Even where this hurdle 
is overcome the status of affective measuring instruments is quite pri- 
mitive. The more successful evaluations of affective gi >wth are tied to 
specific and behaviorally stated objectives, and special instruments are 
designed for the purpose. 

A number of tests and procedures for evaluating affective outcomes 
are di8>-ussed below. In keeping with our general approach, the analysis 
of EEVD ipact on affective growth will: (1) examine affective objec- 
tives as they are represented in curricula, policy, and piograms, to 
determine the schools' attempt to produce these outcomes, (2) determine 
how administrative intentions are actually implemented at the classroom 
level, and (3) attempt to assess student affective gtrowth. 



EDUCATIONAL AFFECTIVE / BJECTIVES 

Schools can respond to incentives and pressures arising from tJie 
EEVD by attempting to improve or modify affective objectives. Information 



O „ See, for example, Virginia Educational Needs Assessment Study, 1970 , 

ERJjC University of Virginia, Virginia State Department of Education, Richmond, 

"™saffla Virginia. 
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about such objectives can be obtained from statements of curricula and 
school policy, and from interviews with school personnel. A s^^cial 
Attitude survey (Delphi) will elso bt used to determii.e objectives as 
voiced by parents and school personnel and attitudes about the schooJ's 
success in meeting objectives. A "successful" EEVD should be accompanied 
by (1) conveiA^ence in attitudes about education objectives araong the 
various populations of individual schools (parents, teachers, specialists, 
administrator." ) and (2) convergence between attitudes about objectives 
and the schools* success in meeting the objectives. 

If objectives within various sub-groups of the community vary 
significantly from those of school personnel, we will want ^o know how 
tliey differ and how the schools attempt to deal with these differ- 
ences wi hin the context of the EEVD. 

AFFECTIVE OBJEJIIVES IN TIi£ CLASSROOM 

School personnel may introduce new curricuJa and programs as part 
of an augmented set of objectives; the important factor to monitor is 
what change actually takes place in the classroom. Teachers have theii 
own attitudes concerning educational affective objectives, and they have 
teaching styles and modes of interpersonal interaction that are often 
extrecrr^ly resistent to change. We must therefore assess the implementation 
of affective objectives as reflected in teacher behavior. Data for this 
analysis will come primarily from classroom observations. 

STUDENT AFFECTIVE GROWTH 

Measures of affective states are either complex (involving qualita- 
tive interpretation of projective tests and experimental manipulations 
involving tasks unlike classroom learning), unreliable, or both. In many 

The purposes and methods of the Delphi technique of attitude esti- 
mation are elaborated in Appendix The utility of estimating pairent 
and school personnel educational objectives extends beyond the affective 
domain, but will be of particular importance here because agreement about 
the importance of cognitive objectives can be more readily assumed. 
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cases the most sensitive instrument for assessing affect is another 
person. An observer can discriminate between unhappy and happy children 
with greater reliability than can affective tests. However, more subtle 
affective statr.s such as self-esteem, achievement motivation, or attitudes 
about bchool are difficult to determine, and cursory assessment is not 
reliable. 

One of the effects of the EEVD on children may be simply to allow 
them (through their parents) to choose school environments in which they 
are happiest. The overall happiness (or mood) of a classroom can be rated 
by observers. The technique for doing this is described below. 

A number of methods will be used to assess student affect. However, 
because of the loW reliability of available methods they will oe used only 
on a sample basis during the first year, in order to identify adequate 
measures. Some of the measuring instruments will probably be omitted as 
the demonstration progresses, and others will be added or modified. It 
would be pointless to continue measurements that show up early in the 
program as insensitive or inappropriate. The primitive state of develop- 
ment of affective assessment makes necessary considerable flexibility in 
this part of the evaluation. Procedures and instruments for affective 
assessment are described below. 

Classroom Sociology 

As parents and children exercise choice in school selection, changes 
in classroom social strv tare and interaction may occur. School policy 
may favor integration, but if students are grouped by ability, de facto 
segregation may result at the classroom level. Within a classroom, teachers 
may produce ja kind of segregation by seating arrangements, or by their ex- 
pectations and interaction with studei*ts. Alternatively, there may be a 
reduction in the number of isolated students fn a classroom, or in con- 
flict involving ethnic and minority groups, because parents select schools- 
on the basis of how "accepted" the child feels in the school. In order 
to assess the impact of the EEVD on the sociology of the classroom, a 
sociogram will be constructed based on the students' response to the 
following questions: 
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1. Name the children in the classroom that you would like to 
inyite to a party. 

2. Name the children in the classroom that you generally play 
with. 

Student-teacher interaction and questions about integration will also 
be investigated through classroom observation described below. 

Student Self-Evaluations 



One reason children dislike school — and some teachers — is their 
feeling that the teacher's opinion of them is low. An attitude scale 

developed by St. John will be used for assessing the child's perceptions 

t * 

of the teacher s opinion of him. This procedure (involving answers to: 

"My teacher thinks I am'*) is described below in the discussion of class- 
room observation, where its use for assessing the effects of teacher 
expectations is discussed. Student self-esteem will be inferred from this 
scale. 

Coopersmith has developed a scale for measuring self-esteem which 
shows some reliability when used experimentally, although responses on the 
test do not discriminate between ethnic and minority groups, nor do they 
appear amenable to change over short periods of time* ' In order to 
further* explore the itility of this test, it will be administered to a 
small sample of students in the first year of the EEVD. The test items 
are shown in Appendix F. 

Sears has designed an instrument for indicating a student's self- 
concept,, and has related self-concept scores :o school achievement. A 
number of research studies report high reliability for a 48 ^^em abbre- 
viated form. This instrument will be administered to a small sample to 



St. John, Nancy, "Thirty-Six Teachers: Their Characteristics and 
Outcomes for Black and White Pupils'' American Educational Research Journal , 
8 November 1971, 635-648. 
** 

Coopersmith, S., The Antecedents of Self Esteem , W. H. Freeman and 
Co., San Francisco, 1967. Private Communication, 1970. 

' Sears, P. S. , and Sherman, V. S., In Pursuit of Self Esteem , Belmont, 
^ California, Wadsworth Publishing Co., 1964. 

ERIC 
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determih^ its worth for assessing status and change In v>elf concept. It 
is shown in Appendix P. The sample chosen will be dilfferent from the 
one for the Coopersmith scale. 

In order to determine the effects of the EEVD on students' attitudes 
about school, a slight modification of a qMestionnaire developed in 
Cincinnati and used in previous Rand work will be used. The two forms 
of the questionnaire, one for grades 1 through 3 (attitudes toward self 
and school) the other for grades 4 through 8 (student survey) are included 
in Appendix F. 

The concept of achievement motivation has been the subject of con- 
siderable research in recent years. It is fairly well established at 
personijlity differences exist between high ar;J low achievers, and thut 
high achievers tend toward learning material that is more structured. 
Traditional mea^'ures of achievement motivation are projective tests, and 
a few attempts: using objective tests have reported low rpli.bility . How- 
ever, Myers reports on an objective test that has reliability comparable 
to that of projective methods. Tais test, shown in Appendix will 
be given in a small sample of classrooms. 

A number of other instruments and prccedures could be used but their 
expected contribution to the evaluation is low and care must be taken not 
to overburden the student vith tests. For this reason, general personality 
tests will not be used at all. They are time consuming, they have low 
reliability, and significant changes in scores can rarely (if ever) be 
associated with programmatic changes? in education. 

The procedures discussed in this section, coupled with data from class- 
room observations and community surveys, will provide an adequate data base 
from which to assess the effect of the EEVD on student affect. However, 
because of the state of development of affective measurement, much of this 
effort is developmental in character, and is expected to undergo change 
as the demonstration progresses. 

Rapp, M. L. , Brunner, G. L. , and Scheuer, E. M. , An Evaluation Design 
for San Jose Unified School Dlstrlct'^a Compensatory Education Progfam , The 
Rand Corporation, RM-5903-JS, May 1969. The questionnaire was originally 
developed by Cincinnati (Ohio) School District for use in compensatory programs. 

Myers, A. E., Risk Taking and Academic Success and Their Relation to an 
Objective Measure of Achievement Motivation , Educational Testing Service, 
RB-64-2, January 1964. 
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CLASS ROOM OBSERVATION 
Purpos e 

Observation will be carried out In the classroom so that the 
procesaefiof education may be related to educational outcomes. Two 
purposes are served by observation: (1) program varlf Icatlor^ to 
ascertain the congruence between the program as described and the 
program as Implemented, and (2) measurement of tec her effect on 
frtudents' cognitive growth^ as one of the educational variables of 
a student 'p performance. As a measure of tb * educational Impact of 
the voucher demonstration, classroom observational data can help 
relate what happens In the classroom to acr,lons taken by pare^its In 
transferring their children. 

Program Verification 

In order to understand the effects of education, we must under- 
stand the environment in which formal education occurs. One of the 
questions addressed by the t-yaluation is the Imprct of the EEVD' on 
the improvement and diversification of educational programs. A program 
is a set of activities requiring resources^ designed to meet a stated 
objective. The instructional strategy specifies how the resources are 
to be used. Often, however, there is no observable miitch between the 
way a program is designed to operate and the way it is implemented in 
the classroom. 

As a check against the mistaken attribution of given results to 
the effect of a specific treatment, we need a description of the program/i 
as they are designed, and an opportunity to see what Ib actually happen*- 
ing in classrooms. No elaborate observation schedules or checklists need 
be constructed for this purpose, because the nature of each obseirvatl "^n 
is highly dependent on the official description of the program. 

First, a dekjcrlptlon of each program will be obtained from tho 
appropriate school administrator. Th!li description should Include 
information about adult/student ratio, desirable staff characteristics^ 
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instructional technique and material, other resources, and special 
training. With this description in mind, each classroom should then 
be visited and the teacher informally interviewed. The observer will 
then be in a position to describe what actually happens in the clr'is- 
room, and to guard against the pitfall of attributing results to liie 
wrong causes* 

Teac h er Effec t 

A grfia'c deal of recent research on teacher behavior hv^s centered 
around the concept of the self-fulfilling prophecy — the idea that a 
teacher's expectations will strongly influence student performance. 
Teacher expectction will not be measured directly, because the ,^ry act 
of having to r.^spond to questions about what she expects from her students 
can have an unmeasurable effect. Teacher expectation will be measured 
by the way in which it is reflected in her classroom behavior. Special 
attention will be paid to whether that behavior is the same toward fill 
students In the class or different for sub-sets of students. 

Much G.t the research on teachers strongly suggests the futility of 
collecting data in the form of self-adtiiiuistered tests to determine 
their attitudes.* The literature on the unreliability of self-report 
inventories is extensive. Studies have shown that some widely used 
inventories are highly susceptible both to faking and to spurious self- 
description. Given different sets of instructions, the same person will 
make two different scores on succt'ssive administration of an inventory. 
This again suggests the need for classroom observation. 

Since one of the desiderata of an evaluation is that it be unobtrusive, 
the task of the evaluator is to choose the least number of measures that 
promise the hij^hest probability of accomplishing his objectives. In 



Donaldson, T.S., An Information System for Educational Management, 
Vol III; Data Requirements for Evaluation; A Review of Educational 
Research . The Rand Corporation, R-932-LACS (forthcoming). 
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addition,, when an observation schedule is constructed, one of its 
salient chararteristics is the degree of inter-rater reliability. A 
complex schedule requires intensive training of highly shil^.ed observers, 
since it generally encompasses many dimensions of behavior to be 
described. If, however, the evaluator specifies few dimensions, and 
each one is ameui^'ble to unarr.Mguous observation, high inter-rater 
rM lability will be acliievetl witli reJatlvely little training of observers. 

In order to meet the criteria of being unobtrusive and at the same 
tjme show good promise of high interpreter reliability, a shortened 
version of Ryan's Characteristics of Teachers Scale as modified by 
St. John v^iil be us^d. The three items on which there was lowest 
inter-rater reliability will be omitted, as will an overall score, 
resulting in ten aspects of teacher behavior to be rated on a seven 
point scal<=i: 

Aloof- responsive 

Dull-stimulating 

Partial-Fair (racially) 

Unsympathetic-Understanding 

Harsh-Kindly* 

Uncertain-Confident 

Disorganized-Systematic 

Inflexible-^Adap table 

Pessimistic-Optimistic 

Narrow-Broad 

Items with high factor loading on child-ojriented dcale. 

Items with high factor loading on task-oriented acale. 

Each of these attributes will be described in behavioral ter^is when 
instructions to raters are written. Sufficient guidance to ensure 
comparable use of the rating scale will be given, but a tightly 
constructed set of directions will be avoided. Too much detail in the 
specification of behaviors to observe creates too narrow a focus for 
observers. 
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II should be recalled that the design of t'le evaluation specifies 
that it will be flexible and evolving. Therefore, out intent is to 
observe in the first year a random sample of teachers (stratified to 
insure representation o*^ all grades and schools) using the scale just 
outlined. Each teacher selected for observation will be observed for 
a total of six class hours during the academic year. In this way a more 
representative sample of her behavior is obtained than if she Is 
observed, say^ for one day. In six different observations, mere 
opportunity is availr/ble to see a vari^ity of cl*.xssroom lessons -and 
teacher behavior. Whereas a teacher's behavior may not be representative 
during onr. observation because of the presence of an observer, this 
effect will be lessened with repeated clasflrootn visits by the same 
person. Furthermore, explaining to the teacher the purpose of class- 
room observation and emphasizing that her performance is not being 
evaluated will greatly mitigate her natural apprehension. The analysis 
of the scale will be designed to relate the observed teacher characteristics 
to student outcome, as measured by cognitive tests. 

Scores will be derived for each teacher on both the child-oriented 
and the task-oriented scales. Each classroom in which a teacher was 
observed will be stratified on the basis of students' entering achieve- 
ment scores in reading. Three groups of students will be formed — 
those whose scores were high, average and low In relation to the mean 
entering reading score for their classroom. For each group of students, 
a gain score from pre- to post-test will be computed. The distribution 
of gains will be examined in relation to the teacher's observed behavior 
scores to see if there is a measurable difference in student outcome that 
can be attributed to teacher behavior. The same kind of analysis will 
then be done for individuals. 

A second taeasure of teacher Influence will be obtained by adminis- 
tering to students a short (10-item) scale **My teacher thinks I am," 
also developed by St. John. Whereas observation will provide an ob- 
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jective indication of how teacher characteristics influence student 
performance, it is also necessary to obtain a rseasure of the child's 
perception of his '*worth" in the eyes of his teacher. 

The St, John scales ask each child to rate himself from 1-5 on 
how he is perceived by his teacher on the following dimensions: 

sad-happy 

lazy-hard working 

mean-kind 

proud-not proud 

stupid-smart 

bad-good 

uasure-sure 

unsuccessful-successful 

not a good student^a good student 

foil6wer- leader 

A consideration of both his teacher*s actions and a child's 
perception of himself in the classroom should provide a reasonable 
explanation of a student's performance. 

CLASSROOM SAMPLING SCHEME 

The purpose of the classroom sampling scheme is to make feasible 
the use of a fairly large number of different tests and measures without 
overburdening the student, test administrators » or data analysis. Tests 
and procedures which are high in reliability are given to large samples, 
while those lower in reliability (or untested) are given to smaller samples 

A school district contains many schools, each comprised of various, 
but not equal, grade levels. Within each grade there are a number of 
classrooms. Since we do not know the school district that will participate 
it is impossible at this time to determine the exact nunber of classrooms 
for each grade at each school. In the final application the sampling 
scheme will have to include considerations of SES and other factors to 
insure that pertinent factors are sampled. However, since this informa- 
tion is not now known, the sample procedure is an approximation, and is 
based mostly on considerations of relative,, rather than absolute, sample 
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slze. In order to arrive at these approximations — especially for 
the time-consuming measures used on the smaller samples — we considered 
a school district of 15,000 students and 40-60 classrooms per ^rade 
spread over thirty elementary and Intermediate level schools • The 
sample structure presented is for the first year of the EEVD, in later 
years refinements in evaluation procedures and test selertion will 
alter the sample plan. 

Th(2 sample plan is shown conceptually in Table 1II-3. Tlie s tiindardized 
math and reading tests will be given to all classrooms. 

The basic purpose of the tests showri in Table II1-3 is to allow an 
assessment of a wide range of student performance. Some of these 
performances have the potential to change rp.latively rapidly over time 
(math and reading), while others will not (self-concept, mental 
abilities, etc.)- Tests measuring those activities which are more 
amenable to change will be given on a pre- and post-basis. These tests 
are Indicated in the first column of Table ly and consist of the math 
and reading standardized tests, and the scaling procedure f o , classwork. 

Those performances which are expected to change slowly (if at all) 

over time will be administered once a year in the spring, Ht^ginning in 

1973. Thus, the first test administration occurs before the EEVD begins, 
and is part of the baseline data. 

The sampling scheme not only allows for an assessment of achievement, 
but Interrelationships between test scores can also be investigated. In 
some cases a test score may show no achievement change » but is useful in 
the interpretation of other test scores. For example, it will be possible 
to analyze reading achievement in terms of basic ability, self-concept, 
creativity, and other scores, rather than simply in terms of the whole 
classroom taken as a homogeneous body. 

Affective growth and reasor\ing ability are not expected to change in 
a short time and it will be nectcisary to fellow students over several years 
in order to detect possible changes in these factors. Since these tests 
are not given to all students, and because there is mobility in and out of 
the school district, it may not be possible to follow adequately the first 
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Table III-3 

DATA SAMPLE STRUCTURE^ 
Data Source Grade 
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Scaling of Performance 
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Soc logram 


SP 


d 


d 
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d 
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Classroom Observation I 


TERM 


a,b 


a,b 


a,b 


d , b 


a,b 


a/b 


a^b 


a,b 


Classroom Observation II 


TERM 


d 


d 


d 


d 
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Maximum tests for group a = 
Maximum tests for group b = 




3 
3 


3 
3 


3 
3 


4 
4 


5 
5 


5 
5 


6 
6 


6 
6 



a and B are independent samples of classrooms, c is a sample made up of class- 
rooms in sample a and b, and c = l/4a + l/4b. Sample d = l/2c. See text for details. 
2 

P-P = pre-post (Fall, Spring)* 
SP = Spring only 



year sample. The sample size is small on some measures, and in the 
later years it will decrease as students move out of the district. 
However, those who remain in the district will be tested in subsequent 
years. 

Analysis of test results after the first year should produce a 
smaller test battery for use in subsequent years, and will be 
administered to a much larger sample, perhaps all students. While 
the mobility problem will still exist, the diminishing sample problem 
will be less serious , 

DATA ANALYSIS 

We have said very little about specific data analysis methods. In 
many cases the meLhod is obvious; in others, it may not be so obvious. 
The use of change versus status scores has already been discussed. In 
those cases where change scores are used (if they are) the regression tech- 
nique suggested by Cronbach and Furby (1971) will be used to estimate 
true change. 

The various student inventories (self-concept, self-esteem, or n- 
achievement) will be analyzed in terms of their subscores, and statistical 
tests wil?L be used to determine the significance of changes in the 
proportion of students in subscore categories. - We do not anticipate 
factor analysis of these data. Other investigators have already factor 
analyzed these tests and the factors (and test items related to each 
factor) are reported. 

Multiple correlation analyses will be used to investigate, the relation- 
ships between the various achievement indicators. If it appears necessary, 
we will investigate the multidimensional achievement space through factor 
analysis. 

The classroom observational d^^ta will be analyzed using counting 
and sorting techniques and appropriate statistical tests of significance. 
Snae content analysis may be attempted in addition. Analysis of school 
records and interview material will be primarily in terms of content. 
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In general, the key analytic question is not statistical, but 
rather how to sCratify, classify, and compare various measures to 
that meaningful interpretations can be made. 

Table III-4 presents the outcome dimensions, indicators, data 
sources, data collection methods, primary data collection responsibil- 
ity, and preferred data analysis techniques for the assessment of edu- 
cational outcomes. 
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IV. OUTCOMES AKD P0LIC7 

The presentation of Rand's evaluation plan lias moved from the 
general to the specific, including: (1) an introductory overview of 
the structure and approach of the Plan; (2) presentation of the basic 
organizational framework of the Plan— theory and major issues of the 
evaluation and of public policy, specification of categories fur gath- 
ering and ordering information about prcgram outcomes, selection of 
the range of outcome dimensions of initial interest and a matrix shov^- 
ing relationships between information categories and outcome dimensions; 
and (3) extended discussions of the salient issues of data collection 
and analysis, organized according to professional skill areas, with 
tables showing the indicators selected for the study of each outcome 
diraension, probable data sources and data collection methods, and ap- 
propriate techniques for data . analysis . 

In this Section we return to the general level — addressing first 
the major policy questions which have significantly shaped our substan- 
tive planning and next the operational design by which our empirical and 
judgmental data will be translated into informed statements responsive 
to those policy questions. 

_POLICY QUESTIONS 

The basic policy question to which the EEVD evaluaUon must respond 
is, should the voucher mechanism be extended to other c o mmunities? A.s 
we noted earlier, this question requires evaluators to identify effects 
of the voucher mechanism on private and public interests. Schools and 
governments (local, county, state, federal) are the most important loci 
of public interests; families, coram-onity groups, churches, and business 
groups (existing or new) are important loci of private interests. 

Tlie remaining policy questions derive from the first: 

o How desirable is it to implement some mechanism 
that gives parents a more direct voice in choosing 
the schools their children attend? 
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o How should educational diversity, especially 

the creation of new schools, be encouraged by 

public policy, if at all? 
o Should some form of public support for private 

and parochial schools be initiated, and if so, 

what form should it take? 
o To what extent should "marketplace" incentives 

be introduced into education , and what form 

should such incentives take? 
o To what extent are vouchers and their implemen- 
tation in the EEVD a necessary and sufficient 

device for attaining the objectives of public 

policy? 

Changed Options for Parents 

In ascertaining the desirability of giving parents more direct 
control in choosing their children's schools, it is important to recog- 
nize that "desirability" in this instance means different things for 
different groups involved. A positive answer to this policy question 
requires several conditions. First, significant numbers of parents 
must find the voucher mechanism an agreeable form of control, the 
exercise of which brings increased satisfaction with schools. Second, 
parents must exercise thel'r more direct voice in schools in ways that 
educational personnel find acceptable and from which students benefit. 
Third, parental preferences and children* s needs must be sufficiently 
similar to prevent endless division among schools. 

Of course, these conditions are not likely to be met among all 
groups to the same degree at the same points in time. For example, we 
expect parent expression of educational preferences to escalate during 
the early years of the demonstration and then diminish as new patterns 
are established. During the escalation period, expression of parent 
dissatisfaction with schools may increase precisely because they are 
using new options and attempting to express preferences. With respect 
to the second condition, it is possible that school personnel and 
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parents will develop agreements about education long before impacts on 
students appear in aggregate measures. With respect to the third con- 
dition, special concerns of different ethnic and income groups may 
surface heretofore latent conflicts, with increased benefits to some 
groups interpreted as disadvantages to other groups. 

For these reasons, the evaluation reports will give more credence 
to relative measures and patterns of congruence between elements over 
time than to absolute measures of particular elements at given points 
in time. 

Educational Diversity 

Should public policy consciously promote educational diversity at 
all? If it should, is the preferred method: ' 

o To increase diversity within the existing public 

school formula? 
o To support the establishment of new schooling options 

outside the public schools? 
o To support both options? 

The case for educational diversity will be addressed in operation- 
al rather than normative terms. If under the EEVD a substantial number 
of parents choose to send their chidren to schools with widely dif- 
fering characteristics relative to the prs-voucher situation, then we 
might conclude that educational diversity is actually sought in 
practice- 

Many observers have expressed concern that support of* diversity 
through vouchers will lead to the decline or disappearance of the 
public school system, \^±th a number of potential disadvantages: 

o The growth of racial and class segregation in 

the voucher-financed schools, reflecting parental 
pref eiencest 

o The relegation of public schools to the position 
of educators of last resort for problem pupils or 
handicapped pupils that the voucher schools reject. 
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o Consequent to such trends, the unifying social 
role oE the public schools in a democratic 
system will disappear, while the voucher schools 
substitute a separating social influence. 

While the EEVD is designed to prevent segregation by race or socio 
economic class, there can be no assurance that this inttint will be met. 
The evaluation reports will display information about the actual distri 
bution of student applications and enrollments by race, Similarly, the 
resulting composition of student bodies in public and private schools 
will be displayed as evidence concerning the flow of children with 
handicaps, educational disadvantages, and disciplinary problems to 
public and private schools in the demonstration area. 

The EEVD cannot provide a final answer to the question of whether 
a voucher system significantly reduces political democracy or social 
unity. The information on school segregation by race and class will 
provide partial evidence. Additional information will flow from the 
parent and community surveys, community observation, interviews, etc. 
But it is entirely possible that a five to seven year period is not 
long enough to provide conclusive evidence.. What is taken in the short 
run to be clear indication of systematic social or political fragmenta- 
tion may, over a longer period, turn out to be an adjustment to new 
situations, with no significant long run consequences for social unity 
or political democracy. Similarly, no major effects may be noted 
during the demonstration period, but over the long run the new institu- 
tions may reduce communication between various social groups. 

The Religious Issue and Support to Private Schools 

Public support of parochial schools has been a Icag-standing 
source of conflict within education and at every level of government. 
Public officials and citizens will ask whether the EEVD escalates or 
dampens these conflicts. 

If the EEVD were to cause the existing education system to become 
substantially more fragmented along sectarian lines, it is likely that 
the religious issue would become more severe. People might perceive 
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the EEVD as a mechanism for providing public tax funds to support 
religion. In this case, the EEVD would escalate rather than diminish 
community conflicts. 

It should be remembered, however, that the present education 
system already contains a sizable sector of schools sponsored by reli- 
gious groups. Pareuts of children in these schools have become 
increasingly restive about the financial burden of school taxes in 
addition to tuition. The increasing cost of operating these schools is 
leading to actual or potential closure of religious-sponsored schools 
and transfer of educational responsibilities for their students to the 
public schools. Therefore, political demands for public aid to paro- 
chial schools are increasing. Conceivably vouchers could be a way of 
meeting this demand \^ithqut violating the Bill of Rights or local 
political feelings about religion and the schools. 

The evaluation of EEVD must probe the policy implications of the 
religious issue. There are several pertinent EEVD outcomes. One is 
the extent to which religiously oriented schools increase enrollments 
and obtain financial support from the EEVD. Another is the attitude of 
conununity groups and community leaders towards such changes. Still 
another is the attitude of the parochial schools themselves towards the 
EEVD. Finally, there are the attitudes of the individi^al members of 
the community toward parochial schools. Pertinent data will be obtained 
from a number of ASC data sources, as described elsewhere in the Tech- 
nical Analysis Plan. 

Public support for private schools does not raise the constitutional 
issues that support to parochial schools raises, but it still generates 
political concern. Will there be a shift of funds for education for the 
broad community to education for a social or economic elite in private 
academies? 

The analytical task will be to determine if such a shift has taken 
place. The rules of the EEVD are designed to insure that any increase 
in enrollment at private schools allocates spaces among all income 
groups in the community. Whether this plan works out in practice Is 
important information for public policy. 
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If some of the private schools are sponsored by political groups 
and particularly if the political groups involved are at one or the 
other end of the spectrum of political ideology, the question of indoc- 
trination will become a political issue* The possibility that tax 
funds would be used to further noneducational objectives is sure to be 
discussed • 

Racial segregation questions are also sure to arise. Again, the 
EEVD rules seek to preclude political or racial exclusivity, but their 
effectiveness must be analyzed. 

Data illuminating these issues will come from several sources. A 
particularly important source, however, is observation of schools and 
analysis of their programs. 

The Harmony of the Profit Motive and Educational Objectives 

Voucher systems seek to harness the desire for private profit to 
the service of improving education. Are these compatible? Compatibil- 
ity depends upon two factors: (1) parental information about education 
(2) the congruence between parental preferences and the educational 
needs of their children. 

If parents have or can obtain sufficient, objective and correct 
information about (1) the educational needs of their children, (2) 
available educational processes and (3) potential educational results, 
then it is likely that the desire for profit will be consistent with 
i iUcational objectives. Informed parents will be able to select 
rationally from among suppliers of educational services. The actual or 
would-be supplier© of educational services must be prepared to offer 
informed buyers quality seinrices and deliver what they promise or 
suffer loss of students to other schools. If, on the other hand, 
parents are uninfomed, conditions will be propitious for fraud. Huck- 
sters seeking a fast profit can make offers that cannot be fulfilled, 
but parents may be unable to perceive this because of their lack of 
knowledge. If parents do perceive this failure, the promoters may 
simply move on to some new field with their profits. In such a situa- 
tion a form of Gresham's Law would apply and honest schools would tend 
to be driven out by shoddy operations. 



-105- 



This form of market failure might lead parents to learn by 
experience so that in later years they would have the information re- 
quired to demand meaningful promises, make perceptive choices and 
require high standards of entrepreneurial performance. It is vital, 
therefore, that the Analysis and Survey Contractor assess not only how 
adequate the informational quality of the market is at the start of the 
EEVD, but how this dimension of the market changes throughout the 
conduct of the demonstration. 

In sum, one important determinant of the harmony between the 
profit motive and educational objectives is whether parents are or can 
become well informed about education. If not, market failure is likely 
and the profit motive will lead to educationally adverse results; if 
they are informed the desire for profits should be a force for 
educational improvement . 

The parental surveys give extensive attention to parental informa- 
tion. Various aspects of the empirical field research will explore the 
program (product) offerings of schools and information generating 
activities. It will be the task of the public policy analysts to put 
the various sources of information together in order to assess the 
informational adequacy of the education market in each demonstration 
site. 

The second aspect of profit-motive and educational-objectives 
harmony is more complex. In the EEVD, parents are the purchasers that 
select schools. It is Lheir children, however, who actually receive 
or consume the educational services. Are parental preferences for 
education congruent with the educational needs of their children? 

Under present educational arrangements, the definition of educa- 
tional needs and selection of programs is basically the responsibility 
of professional educators. School boards are lay-controlled; parents 
have inputs of various sorts in determining programs and curricula; 
and affluent parents may be able to move to different school districts 
or put children in- private schools and so influence the educational 
programs provided their children.. In these ways parents can exert 
preferences among available educational offerings. Even so. 
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professional educators have a major if not dominant role in assessing 
educational needs, selecting curricula and determining programs. 
Teachers' professional training and experience is supposed to give them 
expertise in these matters that lay persons lack. 

The EEVD will shift the balance in the determination of curricula 
and progranis away from professional educators in favor of the laity. A 
clear public policy issue is the effect of this shift. 

I 

Rand's analysis of this issue will center on an examination of the 
consistency or disparity between parental preferences and goals for 
their children's education and the standards of the education profes- 
sion. If there is a high degree of consistency then the significance 

! 

of this issue diminishes. If there is a substantial disparity then it 
becomes very important for public decision. Evaluators, of course, 
cannot determine whether parental desires or professional standards 
should prevail; this is a normative judgment beyond the realm of 
analysis.^ The evaluation^ however, should be sensitive to the potential 
disparity between what professional educators believe to be the educa- 
tional needs of children and how parents perceive these needs. 

Again, the time pattern of change is very important. Data relevant 
to the congruence question will come from several sources, as shown else- 
where in this report, but particularly important sources of data are 
parental surveys and professional observation and analysis of school 
programs. 

Alternatives to Vouchers 

The public policy concerns discussed above are serious enough to 
warrant asking whether vouchers are the most appropriate method of 
meeting public policy goals. For example, we have pointed out above 
that one might seek to promote diversity, choice, and parental control 
through changes in the public schools rather than changes in the 
educational marketplace. 

The evaluation of EEVD, taken as a whole, should cast considerable 
light on this question. If the evidence shows either that the 
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objectives of the demonstration are not being met, or that they are 
being met at high costs in terms of the public policy issues discussed 
above, then other methods of meeting EEVD goals may become imperative, 
if they are to be attempted at all. - 

The evaluation will also offer evidence about the nature of alter- 
native systems. For example, if it turns out that the EEVD public 
schools offer substantial diversity in curriculum as compared to other 
public schools, then a workable alternative may be open enrollment 
plans which include incentives for school administrators to respond to 
market demand. 

More generally, the evaluation data can be used to indicate 
whether the important favorable outcomes of EEVD are separable from 
whatever adverse consequences may result from the voucher experiment. 
This would allow policymakers to decide whether they consider vouchers 
a necessary condition for meeting EEVD objectives. The basic function 
of the evaluation of the demonstration, of course, is to determine if 
vouchers are a sufficient condition for meeting EEVD goals. 

ORGANIZING THE PUBLIC POLICY ANALYSIS 

The basic tasks of data collection and analysis will be performed 
by three professional teams. Each team (political/social, economic/ 
cost, education) will test the utility and relevance of data to be 
collected, and tha feasibility and reliability of the various analytic 
techniques proposed. Each team will assign data collection and 
analysis priorities as the demonstration proceeds. The team's work 
will reflect an understanding of the salient characteristics of the 
EEVD and the special requirements for evaluation associated with large-- 
scale social demonstrations (summarized in Section II, above). In 
particular, a flexible and adaptive posture will be .maintained so that 
the public policy questions and changed policy priorities can be 
reflected in the evaluation. 

The first aggregation and analysis of data ill support of detailed 
empirical generalizations will Dlnvolve statements about the indicators 
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which have been selected as measures of the various outcome dimensions. 
This level of aggregation will be the responsibility of the profes- 
sional teams. 

These teams ^will also conduct the next level of data aggregation. 
For each outcome dimension, relevant indicator findings will be 
inspected and assessed in light of experience with the data and with 
problems of analysis. In each case, a descriptive statement will be 
formulated that elaborates key findings. An example of a format that 
could summarize these findings is shown in Figure* IV-1. 

The next step in the derivation of successively higher levels of 
empirical generalization will be the inspection of evaluation findings 
across outcome dimensions . This analysis , and all subsequent analyses , 
will be the responsibility of the project senior staff, composed of key 
professionals from each of the task teams plus the project director and 
his key deputy. 

Figure 1-3, p. 16 > illustrates how the findings on program out- 
come dimensions will be aggregated according to selected information 
categories, which serve an accounting and organizing function. Reading 
across the rows of this matrix, we may now list the outcome dimensions 
that will.be considered in each of these categories. The outcome 
dimensions that will be treated have been shown previously and are 
listed in a convenient form in Appendix G , Findings will be taken 
directly from each of the individual reporting forms for outcome 
dimensions as illustrated in Figure IV-1. ASC will assess the rela- 
tive importance of each outcome dimension in formulating relevant 
conclusions in light of demonstration experience, and will draft a 
report summarizing key findings suggested by these comparisons. A 
format for this purpose is shown in Figure IV-2. 

Finally, similar procedures of aggregation, inspection, weighting 
of importance, and formulating conclusions will be applied to the 
findings summarized in each category of information in order to atrive 
at overall conclusions related to each of the major questions of the 
evaluation. To reiterate, these questions, as well as the information 
categories, outcome dimensions, and indicators, will almost certainly 
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be refined and amended in the light of actual experience with the 
forces and processes of the demonstration. We have illustrated the 
relationship of findings in each category of Information to major ques- 
tions of the evaluation, in Figure 1-2, p. 13, Reading across the 
rows of that matrix, we may list the information categories that will 
be assessed in order to answer each of these quest±ons« Table lV-1 
presents this list. 

At this level of aggregation, even summary statements will be com- 
plex and detailed; their presentation will not readily yield to the 
type of format suggested for the display of summary findings at lower 
levels of empirical generalization. Accordingly, findings will be pre- 
sented in a less tabular mode. 

The Need for '*Fine Grain" Findings 

The formats for the display of summary findings at each stage of 
the analysis are extremely detailed. Some might prefer thaC: the 
evaluation report supress the detail in favor of aggregative summary 
findings* We propose to provide generalized and summary displays and 
discussions* However, we emphasize again the importance of maintaining 
and presenting a disaggregated and fine grain description and explana- 
tion of the demonstration. Findings at all levels of generality will 
have implications for public policy; the nature of these implications 
will be determined by the specifics of the policies under consideration 
and the unit of government involved, whether at the federal, state, or 
local level. There will not be a smooth transition between successively 
higher levels of empirical generalization and the ability to draw public 
policy inferences; this end is best served by being able to inspect 
statements about demonstration outcomes from the mixed perspective of 
various levels of generalization regarding program findings , 

POLICY OPTIONS AND INSTRUMBNIS 

The objectives and the range of possible outcomes of the voucher 
demonstration are multiple and complex, but the tools available to the 
policymaker are comparatively few in number, are rarely refined, and 
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Table IV-1 

EVALUATION QUESTIONS AND INFORMATION CATEGORIES 



QUESTIONS 



CATEGORIES 



Impact of EEVD on! 



Education of elementary school 
students 



Range of choice among educational 
programs 



Equality of educational opportunity 



Economics of public education 



Citizen-school relations 



Critical social and political 
tensions 



Education Results 
Attitudes of practitioners 
Programs and processes 
Attributes of new schools 
Allocation of resources 
Financial impact 

Consequences Leyond demonstration 
area 

Programs and processes 
Attributes of new schools 
Distribution of stude'^.ts 
Consequences beyond demonstration 
area 

Educational Results 
Attitudes of practitioners 
Programs and processes 
Attributes of new schools 
Distribution of students 
Allocation of resources 
Consequences beyond demonstration 
area 

(Allocation of resources 
Financial impact 
Consequences beyond demonstration 
area 



/ Governance and administration 
I Status of professionals 
/Parent attiltudes and responses 
•^Community attitudes and responses 
I Consequences beyond demonstration 
\ area 

I Governance and administration 
Status of professionals 
Parent attitudes and responses 
Community attitudes and responses 
Consequences beyond demonstration 
area 



must be applied in complex and fluid social settings where the conse- 
quences of a given policy application are rarely predictable with high 
confidence- In addition, policies — or the means by which policies are 
to be realized — may be in conflict with one another. The multiplicity 
of public and private interests that may be affected by any given 
policy application implies that there is not likely to be any way of 
coming closer to the goals of one public policy without moving further 
away along another dimension, and that policies of "social optimization" 
will be impossible to devise. 

These considerations emphasize the importance of securing a careful 
understanding of the range of public policy options and instruments that 
may be available, and of relating them in some systematic fashion to the 
findings of the evaluation. If reliable implications for public policy 
are to emerge from the evaluation of the EEVD, analysis must not be 
divorced from the decisionmaking process . Accordingly, this evaluation 
plan includes the following three procedures. 

First, Rand will work closely with appropriate agencies of federal, 
state, and local government in order to clarify the public policy issues 
of concern, define the policy options that may be considered, and trans- 
late these policy options into operational terms. It is anticipated 
that this work can be of benefit both to policymakers and to evaluat'^rff. 
For policymakers, it can help to focus attention on the range of possi- 
bilities suggested by the demonstration, and give them time to consider 
the various alternatives in which they are interested. For evaluators, 
it can provide useful feedback from the policymaking community regarding 
the foci of public con'Siern, and assist in the setting of priorities fo^ 
data collection and analysis in the course of the evaluation. 

Second, senior Rand evaluation staff aembers will monitor evalua- 
tion findings as they grow in detail and in levels of generality, and 
will attempt to define and describe evailuation outcomes that could be 
regarded as evidence in support of public policies. A range of policies 
will be considered, and a list of potentially feasible options will be 
drawn up, together with the operational implications associated with 
each option. This list will be refined and revised in the course of 
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continuing discussions with policymakers at all levels of government. 
For each policy option on the list, actual and possible demonstration 
outcomes that might provide evidence in support of a policy implementa- 
tion decision will be elaborated, and the reasoning underlying their 
selection will be detail?ed. 

Third, as the evaluation progresses and Rand staff come to have a 
growing understanding of the ou(:comes of the demonstration that appear 
probable, senior evaluation staff will reinspect the available data 
and the findings on outcome dimension and their indicators, in order 
to specify the conditions under which there would appear to be a 
reasonable chance for- the replication of each outcome of interest to 
policymakers. The conditions of interest will include essential prog*- 
ram ideas or components that must be applied, as well ay the general 
dimensions of pollti^cal, social, economic or educational conditions 
under which the application of such program components seem most likely 
to achieve desired results. Here, in particular, it will be essential 
to derive conceptual equivalence rather than operational equivalence as 
a guide to policymakers interested in program or program component 
replication. 
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V. MANAGEMENT AND PLANNING 

This section describes the major elements in the organization of the 
Phase II analysis and survey effort, including: 

1. Organization of the work. 

2. Staffing of ASC effort. 

3. Schedule of work and products. 

4. ASC relations with other agewcies participating in EEVD, 
including ASC survey subcontractor. 

5. Access to and privacy of data. 

6. Methods of periodic review and adjustment of analytic design. 

The following description is based on a Phase II effort beginning in 
March 197?., consistinv?> of an eighteen-month pre-demonstration period, 
followed by five consecutive one-year demonstration periods to start in 
September 1973 at two to five demonstration sites. If at one or more 
sites there should be a demonstration effort beginning in September 1972, 
the pre-demonstration schedule described below would be compressed from 
eighteen months to six months for the early start sites. For other sites 
the eighteen-month pre-demonstration schedule would continue to bp 
observed . 

RAND ORGANIZATION AND STAFF 

Rand's organization for Phase II of the EEVD analysis and survey 
contract would be as described in the Rand Phase I proposal of September 
1971, with some modification resulting from changes and clarifications in 
the program proposed since then by OEO. 

Introduction: Rand Management Structure, General 

Before describing the proposed Rand organization for Phase II of the 
EEVD analysis and survey, it is useful to describe Rand's management 
structure. 

The Rand Corporation is a nonprofit corporation, incorporated under 
the laws of the state of California, and performs research on national 
policy, strategy, and operations that affect the security of the United 
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States and on domestic affairs including problems of education, health 
care, housing, poverty, and pollution. 

Rand's Board of Trustees presently consists of eighteen members. 
Three of these trustees are officers of the Corporation while the remain- 
ing fifteen iare outstanding individuals from industry, the professions, 
and universities. 

The management structure and organization of Rand have been designed 
to facilitate carrying out interdisciplinary research programs and to 
permit establishment of problem-oriented "centers" and "institutes" (such 
as the New York City-Rand Institute). As Fig. V-1 shows, the Rand 
research staff, numbering about 450, is divided among six research de- 
partments, each of which is specialized in a particular discipline (e.g.. 
Engineering Sciences, Economics) or skill (e.g.. Management Sciences). 
Overlapping this departmental structure is a functional program structure 
(e.g.. Education , Health , Environment) . 

Each program manager is responsible for developing and maintaining a 
program of research and analysis centered upon a major problem area. He 
draws the staff for his program from the technical departments; he is 
directly responsible to corporate management for the quality, timeliness, 
and costs of his program; in him is vested authority and responsibility 
for budgetary control of the program. Typically, the program manager 
directs the efforts of several project leaders, in conjunction with whom 
he maintains liaison with the sponsors of the several elements of his 
program. 

The department head is responsible for maintaining the excellence of 
the research staff; for carrying out basic and background research; for 
conducting research to develop and test new analytic tools and methods; 
and for supporting Rand's program-oriented research. 

The New York City-Rand Institute is a non-profit research institution 
formed primarily to conduct programs of scientific research and study, and 
provide reports and recommendations, relevant to the operations, planning, 
or administration of the city of New York. The Institute was established 
in 1969 as a joint venture by the city of New York and Rand as a center for 
the continuing application of scientifc and analytic techniques to problems 
of urban life and local government. Its program includes work on health 
planning, policy, and delivery; drug abuse, housing; fire protection; 
criminal justice, welfare; economic development; and other city problems. 
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Rand Management: Structure for Phase II 

Figure V-2 shows the proposed Rand organi2ation for Phase II, including 
relations with other ngencies involved in i£EVD. The orgnnlzation (if the 
work will be based on the approach discussed in Section III above, 
reflecting three elements: (1) field monitoring and survey^j; {2) evalu- 
ation of the educational, social/political, and economic effects of EEVJ); 
(3) policy implications of the evaluation. 

There will be two EEVD project deputies reporting directly to Rand's 
program manager for education, who will devote substantial time to the 
project. 

In each demonstration city, there will be a site director, responsible 
for all field monitoring and analysis in that city. He will conduct much 
of the observation and data gathering, and direct and coordinate the work 
of other Rand staff members (including community observers) and consultants 
in that location. He will also be responsible for coordination with EVA, 
ISC, and the survey subcontractor's field director in that location, as 
well as for assuring preparation of reports dealing with his site. 

A group of senior staff specialists will work with site directors in 
developing the field monitoring program and jointly with the site director 
and survey subcontractor on survey design and review issues. One infor- 
mation specialist will be primarily responsible for liaison with the data 
management contractor. 

These staff specialists, with expertise in demography, sociology, 
economics, psychology, education, political science, and systems analysis, 
will also work on the three vpajor evaluation aspects discussed in Section 
III — educational, political/social, and economic. An analysis group 
composed of appropriate senior staff specialists and consultants will be 
responsible for each of the three areas and will receive technical support 
from Rand information science, mathematics, and statistics staff. (See 
Fig. V-2.) 

These team members will work in all of the demonstration districts 
but in each district they will work with and under the supervision of the 
site director. This "matrix'' organization will offer the advantages of 
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specialist e*;pertise, at the same time providing coordination and clear 
lines of responsibility. 

The entire field monitoring and analysis effort will be under the 
supervision of the project director. He will have three prime responsi- 
bilities: liaison with OEO and other participants, coordination of 
on- and off-site efforts j supervision of the staff specialist teams in 
their analysis of the major poli^cy issues, 

A project review board will be selected composed of several Rand 
staff members (Robert Levine, Anthony Pascal, Roger Levien) and several 
other nationally recognized experts on evaluation, drawn largely from 
the panel who were inviued to review Rand's Phase I approach in December 
1971, They included David Cohen (Harvard University), Peter Rossi (Johns 
Hopkins), Henry Levin (Stanford), James Coleman (Johns Hopkins), bheldon 
White (Harvard), Martin Rein (M.I.T,), Richard Snow (Stanford), Robert 
Stake (Illinois), Eleanor Sheldon (Russell Sage Foundation), Alice Rivlin 
(Brookings Institution) , Norman Kurland (New York State Department of 
Education), and Harry Vakos (Minneapolis City Schools). The project 
review board will regularly review the progress of the work, including 
plans developed during the pre-demonstration period and the annual evalua- 
tion report draft. It will advise the project director on technical and 
policy issues as required. This function will include reviewing with the 
project director the need for periodic adjustment of the analytic design 
in light of experience. Finally, the board will itself serve an evaluation 
function by evaluating ASC efforts as they progress. . 

For those demonstration sites and comparison sites (if any) located 
on the Wefet Coast, Rand would plan to keep the bulk of the team together 
in its Santa Monica office and travel for field work. Two community 
observers will be permanently stationed on site. They and the site 
director may require locally recruited research assistance for data col- 
lection depending on the level of information services provided by EVA, 
ISC, and DMC. 

The tactic of clustering most of the project staff for western EEVD 
sites in Santa Monica would facilitate interaction among the study team 
members e For eastern and midwestern sites. Rand will either open local 
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offices or assign staff full time to existing Rand offices in Washington, 
D. C. , and New York. 

The basic organization shovrn in Fig. V-2 can be expanded as required 
to more cities than the two to five now contemplated by OEO. For a large 
number of cities, an intermediate regional office structure could be 
established with offices in two or more principal geographic regions of 
the country. The Field Research Corporation has assured Rand that their 
organizational structure permits substantial expansion of the svrvey 
capability as needed. 

Appendix I lists names and qualifications of Rand staff members who 
would be assigned to EEVD and the functions they would be responsible for 
under the Analysis and Survey Contract. John Pincus, Rand's program 
manager for education, woula continue to exercise general supervision, as 
in Phase I. George Hall and Daniel Weiler would act as project deputies 
and also lead the economic and political/social analysis groups, respec- 
tively. Marjorie Rapp and Theodore Donaldson would lead the education 
analysis group. Barbara Williams would be primarily responsible for 
collaboration with Field Research Corporation, as. well as for collabor- 
ation with Daniel Weiler in leading the political/social analysis group. 
John Farquhar would be primarily responsible for relations with the Data 
Management Contractor, and would also lead the technical support effort 
for information flows. John Rolph would be responsible for the technical 
support effort in mathematics and statistics. Site directors, to be 
selected upon determination of actual sites, will be responsible for deal- 
ings with all agencies at the site level, for data collection and analysis 
pertaining to EEVD at the site, and for supervision of Rand staff and 
consultants on site. Resumes of senior staff are included as Appendix I, 

SCHEDULE OF WORK AND PRODUCTS 

During the Phase I period. Rand staff carried out the seven principal 
tasks described in the September 1971 proposal: (1) validate general 
approach and refine methods, with assistance of (*xpert panel; (2) prepare 
sample design and survey instruments in collaboration with Field Research 
Corporation (see FRC report dated February 2, 1972); (3) develop plans 
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for information flow and processing (see below) ; (A) develop a cost 
analysis plan (see Section III above); (5) structure plans for 
conducting the major analysis tasks (see Sections III and IV); (6) 
structure approach to analysis of policy implications (see Sections II- 
IV); (7) set forth plans for Phase II organization and administration 
(included in this section) • 

During Phase I, Rand has refined and restructured the schedule of 
work and products set forth in the first volume of the Phase I proposal 
(pp. 42-43). The planned work program has been subdivided for expository 
purposes into four sets of tasks: 

o education component 

o economic and resource analysis component 

o political/social component 

o information flow component 

In actual practice, there will be close interaction among the components, 
with outputs from one set of tasks feeding in as inputs to the other sets. 

Table V-1 shows the schedule of work and products for Phase II, on 
the basis of an eighteen month pre-demonstration period and the first of 
five successive one-year demonstration periods. (This information is 
also summarized graphically in Appendix H,) It also shows, for each set 
of tasks shown in the table, estimated Rand professional staff time require- 
ments over the first thirty months, on the basis of a single demonstration 
site. Additional demonstration sites would require a less than proportional 
increase in data analysis time and a proportional increase in data collec- 
tion and observation time. There would be less than proportional increases 
in general management and technical support time requirements. 

The tasks and professional staff requirements shown in the table are 
based on the assumption that a flow of data will be supplied by EVA and 
bhe Information Systems Contractor through the Data Management Contractor, 
In the event that these assumptions are not valid, additional Rand data 
collection and coordination tasks would be required, requiring an estimated 
27 professional man-months during the pre-demons tration period, and 18 
man-months during the first year of the demonstration period for a single 
demonstration site. 



-123- 



Table V-.l 

PROPOSED SCHEDULE OF WORK FOR PHASE II 

(18 month Pre-Demoustration Period 

and first 12 month Demonstration Period) 

Month (nimbered 

Task from start of 

No. Description of Task Phase II) 

1. Political and Social Components 
(including Survey Tasks) 
P/S-1 Pretest and Final Development of Baseline 

Survey Instrument 1-2 

P/S-2 Design and Test Pre-Demonstration Survey Instrument 2-3 
P/S-3 Design and Interview Instruments for Parents, 

Teachers, Government Officials, Community Leaders 1-2 

P/S-A Prepare Training Program for Community Observers.. 2-A 

P/S-5 Recruit and Train ConuTiunity Observers 1-7 

P/S-6 Collect Documentary Historical Data 8-16 

P/S-7 Administer Baseline Survey.... 12-13 

P/S-8 Administer Initial Interviews 8-16 

P/S-9 Organize and Process Documentary Historical Data.. 16-18 
P/S-10 Process Baseline Survey Data and Prepare 

Summary Report 14-15 

P/S-11 Process Interview Data 16-18 

P/S-12 Prepare Portrait of Key Pre-DemonL^tration Trends 

in Demonstration Community 18-19 

P/S-13 Conduct Community Observation 8-30 

P/S-14 Administer Pre-Demonstration Surveys 17 

P/S-15 Process Pre-Demonjtration Survey and Prepare Report 18-19 

P/S-16 Collect Documentary Data.. 17-30 

P/S-17 Code and Analyse Community Observation Data 10-30 

P/S-18 Refine Interview Instruments 19-20 

P/S-19 Prepare Parent/Community Survey //I 20-21 

P/S-20 Administer Parent/Community Survey #1 27-28 

P/S-21 Administer Follow-Up Interviews 26-2:: 

P/S-22 Process Parent/Community Survey //x and Prepare 

Report 29-30 

P/S-23 Process Follow-Up Interview Data 28-29 

P/S-24 Process Documentary Data 29-31 

P/S-25 Aggregate Data on All Indicators and Conduct 

Program-Level Data Analyses 29-32 

P/S-26 Prepare Input to Year-End Report 31-33 

Rand Professional Staff Time Required for P/S Tasks 1-26: 

Months 1-18, 36 man-months 
Months 19-30, 42 man-months 
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Table V-1 — continued 



Month ( nimbeved 



Task from start of 

No. Description of Task Phase II) 

2 . Educational Component 

E-1 Develop Scaling Procedures 1-6 

E-2 Carry out Experimental Scales; Score and Analyze 3-12 

E-3 Field Test Delphi • 5-12 

E-4 Develop Rates Instructions for Classroom 

Observation . 5-P' 

E-5 Determine Test Battery 13-18 

E-6 Collect Baseline Data 15-18 

E-7 Arrange for Follow-Up of Students 17-18 

E-8 Analyze Pre-Test Data 19-22 

E-9 Classroom Observation 19-26 

E-10 Prepare Interview Schedules for Teachers, Aides, 

Administrators 17-18 

E-11 Conduct Interviews 19-26 

E-12 Analyze Post-Test Data 27-30 

E-13 Prepare Inputs for Year-End Report 29-31 



Rand Professional Staff Time Required for E Tasks 1-13 

Months 1-18, 18 man-months 
Months 19-30, 18 man-months 



3. Economic and Resource Analysis Component 
ERA-1 Collect Baseline and Historical Data on Market 

Structure, Behavior and Performance.,,,^ 1-6 

ERA-2 Develop Formats For Educational Programmatic 

Profile Analysis 1-10 

ERA-3 Develop Model for Resource Analysis 1-6 

ERA-4 Develop Model for Analys^ing Funding Flows 1-8 

ERA-5 Collect Baseline Data (Programs, Resources, 

Funding) 7-10,19-22 

ERA-6 Analyze Baseline Data-Develop Progranmatic 

Profile, etc 9-12,21-24 

ERA-7 Test Feasibility of Formats and Models 9-14 

ERA-8 Document Baseline Status 11-14,23-26 

ERA-9 Modify Formats and Procedures 13-16,27-28 

ERA-10 Analyze Organization of Educational Market and 

Competitive Conditions 13-16 

ERA-11 Document Procedures for Tasks ERA 2-4 for 

Operation (input for year-end report) 13-18 

ERA-12 Document and Report Organization of Market 

(input for year-end report) , 17-18 

ERA-13 Collect Year-End Data on Resource Use 17-20,29-30 

ERA-14 Analyze Changes in Resource Use 19-20,31-32 

ERA-15 Monitor Changes in Economic Benavior, Structure, 

Performance 19-26 

ERA-16 Analyze Changes in Organization and Competitive 

Conditions, , , , 27-28 

ERA-17 Docuirir^nt and Report History of Changes in 

Organization, Structure and Behavior of 

Educational Market(input for year-end report),, 29-30 
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Table V-1 — continued 



Month ( mmbeved 



Task from start of 

No, Description of Task Phase II) 

EIlA-18 Document Program and Resource Impact 

(input for year-end report) 31^33 



Rand Professional Staff Time Requirements for 

ERA Tasks 1-18: 

Months 1-18, 15 man^-months 
Months 19-30, 10 man-months 



4 . Information Flow Component 

a. Definition of Base Data Requirements 

IF~1 Interaction and Specification with Research Team 1--18 

IF-2 Prepare Specifications for DMC: Data Names, 

Field Dimensions, etc 1--18 

IF-3 Prepare Collection Specifications (with DMC)... 5--12 

IF-4 Assist DMC in Identification of Priorities and 

Validity Standards 1-4 

lF-5 Interface with Repositories of Baseline Data 

(state, regional and local) 1-3,7-11,16-18 

b . Design and Implementation of Data Accountability 
S ystem 

IF-6 Functional System Design 1-4 

IF-7 Design and Production of Transmittal and 

Notification Forms 4-8 

""F-S Detailed System Design 5-8 

IF-9 Program Coding and Checkout 8-11 

IF-IQ Program Documentation. ... - 10-12 

IF-11 System Documentation 8-11 

IF^12 System Test and Exercise, Using Dummy Data 11-17 

c. Identification and Design of ASC Analytica l 
Tools 

IF-13 Interaction and Initial Specification of ASC 

In-House Aids 1-18 

lF-14 Modify Existing Packages 1-18 

IF-15 Prepare Rapid Data Entry Routines for Rand Data 

Analysis System. » 6-12 

d . Detnoastration Tasks - Year 1 

IF-16 Maintain Data Accountability System: Receive, 

Screen, Issue Receipts, etc 19-30 

lF-17 Identify Altered Requirements, Issue Ch^^nge 

Notices, etc 19-30 

IF^-18 Spot Validity and Comprehensiveniiss Checks 20-21,25-26 

IF-19 Expediting System Change and Coordination 19-30 
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Table V-L — continued 



■hnth (nwnhcVi'd 

Task from start of 

No. Description of Task Phase Tl) 

e. ASC Analytical Assistance 

IF-20 Maintain Computationai and Analysis Aids I9--30 

IF-21 Prepare Additional Tools Required, Listings, etc. 19-- 30 

Rand Staff Time Requirements for IF Tasks 1-21: 

Months 1-18, 30 man-months 
Months 19-30, 18 man-months 

4 . Reporting Tasks 
RQl-8 Quarterly Progress Reports (not issued to coincide 

with annual reports) 4 , 7 , 10 , 13 , ' " >22 . 25 , 28 

RAl-2 Annual Progress Reports (after pre-demonstration 

period, and annually thereafter) 20-32 

RSl-3 Summary Report on Survey Results six weeks after 

administration of each 
survey 



Rand Staff Time Requirements for Reporting Included 
in Analysis Task Requirements. 
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INFORMATION FLOW AND RELATIONS WITH O THER AG ENCIKS 

The basic ASC task in Phase II is to assure that specified data sets, 
quantitative and qualitative, are collected, diiiplayed, and analyzed to 
cast light on the results and policy innplications of EEVD. Assuring the 
appropriate information flow to and from ASC, while guarding the confi- 
dentiality of the data, involves both direct data generation atid proces- 
sing by ASC a?id ASC acquisition and provision cf intormation to other 
agencies. The principal agencies concerned are, in addition to ASC: 

o Office Oi Economic Opportunity (OHO) 

o Educational Voucher Agency (EVA) 

o Information Systems Contractor (ISC) 

o Data Hanagement Contractor (DMC) 

o EEVD technical assistance agency 

o local scJ^ools 

o state school agencies 

o agencies of local general government 

This subsection discusses first the organization for requirements 
and procedures to assure f^hiis information flow, including continuity and 
confidentiality; second* the nature of proposed relations between Rand 
and other agencies involved in EEVD. 

Information Flow: Requirements and Procedures 

This subsection discusses the proposed mechanism for data collection 
and transfer* and describes the measures that must be tak^n by ASC to insure 
an orderly flow of valid information. In that the final organizational 
arrangement of the demonstration i^ yet undecided, th^ expected data flow 
and responsibilities for two eventualities are described: the first pre- 
supposes an established ISC» while the second does not. 

As described by OEO personnel, school and student data will be 6ol- 
lected by ISC, operating under the aegis of the EVA. The ISC may consist 
of a staff retained by t^e EVA, or thes^ functions may be contracted. The 
data collected will be forwarded \to the DMC for preparation, storage, 
analysis, and dissemination to the ASC and 0E0< (Preparation, as the 
responstblllty of the OMC, may occur ^either at the demonstration site or 
at a removed ideation,) 
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Generally speaking, this organizational arrangement may be described 
as a closed loop, with requirements deflnitlo:3 flowing from ASC to DMC to 
ISC, and data flowing In the opposite direction. These relationships are 
Illustrated schematically In Fig. V-3. The effective performance of each 
"side" of the system is critical to the success of the evaluation, and 
will require establishment of formal monlJ:orlng and Inforonation traiisfer 
procedures. Thi£^ is particularly true of the requirements definition 
flow, which is often neglected in consideration of this sort of inf^Jrma- 
tion system design. 

General Approach (Assuming Existence of ASC-Independent ISC) 

The Rand effort in this general area may be divided into fcur over- 
lapping tasks, as follows: 

o Definition of base data requirements (pre-demons tration) 
o Design and implementai:ion of the Data Accountability 

Sys ■ em (pre-demons tration) 
o Identification and design of ASC analytical tools (pre- 

demonstration) 

o Maintaining .^n interfoct with the DMC, and facilitating 
data collection and dissemination. 
Each of these tr^sks is described below, and In Tabls V-1, with a graphic 
nummary shown iu Appendix H. Described below Is the general content of 
each of the four tasks, followed by a discu&slon of the steps necessary 
If Rand must also assume ISC re8poni^ibili$;y . 

Definition of Ba^e Data Requirements . This task will involve con- 
tinued preparation and refinement of the EEVD data requirements, prepara- 
tion of data specifications in a fonn acceptable to the DMC, and partici- 
pation with the DMC in establishing effic^vcnt collection procedures and 
identifying promising data sources. In addition, an important portion 
\0f this task (which will substantially affect th^ role of other parties) 
involves specification of data validity and accuracy requirements, and 
Identification of those areas where evaluation results will be partic\»- 
la^ly sensitive to data validity and Oftaprehensiveness. This exercise 
should lead to a Judicious and well-grounded statement of collection. 
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retrieval and analysis priorities, with respect to both timeliness and 
expended effort. 

Design and Implementation of the Data Accountabij^ity System . The 
problems of commutd cation between ACS, DMC, and ISC are large and complex, 
particularly in consideration of the current flux of demonstration plans, 
aiid the volume and coverafge of require data. We propose to attack these 
problems through design and implementation of a Data Accountability Sys- 
tem (DAS), aimed at providing monitoring and status reports concerning 
the state of both information and requirements definition. 

The DAS will consist of a series of procedures and programs, the 
latter to be implemented upon tha Rand JOSS time- sharing system. Bri<2fly 
stated, the objectives of the system are threefold: 

o to provide, for ISC, DMC, and ASC, a master summary of all 

data items collected, and a digest of their dimensions; 
o to provide an efficient means of cotnmunications — with 
full audit trail — between the ASC and DMC concerning 
collection, dissemination, and requirements definition 
status ; 

o to provide^^ to ASC researchers and OEO monitors > a rapid 
referencv"*. for judgment of the correspondence between 
research objectives and the ability of collected data 
to meet those objectives,* 

Initial definition of the data to be collected has hiien and will 
continue to be established through verbal and written communications 
between the ASC and the DMC. These communications will eventually result 
in preparation of a Master Data Summary, detailing, foi' each data item to 
be collected, the following dimensions: 

o item np/me 

o item description 

o data source (e.g., school, district, state) 

o collection cycle (e.g. , parking-period, monthly, or 

specific event-orientation) 
o collection responsibility 
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o item coverage (specifying, if necessary, par ticular • schools , 
classrooms) 

o PRESS descriptors 
The Master Data Summary will serve as the constant reference and vocabu- 
lary definition for subsequent communications between the ASC and the 
DMC, and a common reference betv?een the ISC and the DMC . The existence 
of such a summary should minimize communication problems between the ASC 
and UMC, and insure that ASC requirements, are accurately reported to the 
ISC. 

Despite extensive efforts at initial specif ication > data require- 
ments may be expected to cliange markedly as the Demonstration (and subse- 
quent evaluation) evolves. Particular areas of interesting activity will 
emerge, certain data items will be recognized as invalid or unnecessary, 
and new sources of Jata will appear. For these reasons, specific pro- 
cedures for alteration and updating of the Master Data Summary must exist. 

The key to such changes is the Item Change Notification, issued by 
the ASC when addition or deletiot? of a data item is required, or when one 
of the dimensions specifi^^ above must be altered. The Item Change 
Notification is sent to tne DMC, along with the revised Master. Data Sum- 
mary. R^aceipt of the Change Notification is acknowledged by the DMC 
through written communication specifying date of receipt and expected 
date of change implementation. Notice of final implementation of re-- 
quired procedural and program changes will be issued by the DMC. Verifi- 
cation of final implementation is the responsibility of the ASC. 

The ASC will maintain the Data Accountability System through estab- 
lishment of an automated system for production of summary lists, m-otifi- 
cations, and receipts. This system will also be used to maintain a 
master schedule for reporting and analysis of data received. Figure V-4 
summarizes the portion of the Data Accountability System dealing with 
requirements definition, and the associated forms and procedures* 

Identification and Design of ASC Analytical Tools . This task, the 
third involving pre-demonstration preparation, requires final listablish- 
ment of the statistical-analytic packages and routines required for 
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effective analysis. This required effort is straightforward in principle: 
the information specialists will confer with researchers concerning their 
methodological needs (usually well-defined), and will marshal and adapt 
the required resources. 

In addition, we intend to adapt Rand's Data Analysis System for use 
by research team members. This system is a Rand-developed aid to users 
of large data bases. In the EEVD , there will often be need to facilitate 
interaction between the researchers and their data base. At the elemen- 
tary level such interaction can decrease the cost and speed up the re- 
searcher's analytic efforts. On a more sophisticated level, increasing 
the interaction of researcher and data provides an opportunity to get 
much more from the data. The researcher can more thoroughly explore 
alternative hypotheses, investigate complex phenomena which do not readily 
lend themselves to straightforward statistical analysis, and, pursue 
hunches and flashes of insight which might ordinarily be forgotten. 

The Rand Data Analysis System aids the researcher in accessing his 
data and assisting him in interactively applying a wide range of analytic 
procedures. He is able to review the raw data in tabular or graphical 
form, add to or delete from the basic data base; flexibly subset, struc- 
ture and restructure the data for hypothesis testing and formulation; 
and apply many of the standard statistical tests. Because tfie displayed 
results and transition between each analytic step are accomplished at 
interactive speeds, the researcher is able to get very close to his data 
by exploring it to a depth wh-'.ch has previously been impractical. 

The prototype system, which is currently being production engineered 
for greater reliability and efficiency, offers a user the following 
capabilities : 

1. Load latge files or subsamples of large files (in a 
batch computer mode) from card, tape or disk; 

2. File raissingLjor undefined data in such a way that the 
system automatically handles it during its computation 
and displays; 

3« Interact with the user from an on-line graphic terminal 
(either Rand videographics or IBM 2250) employing the 
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following display capabilities: 

a. raw data tabular display, 

b. histograms and barcharts, 

c. two-way contingency tables, 

d. m^'ltiple linear regression with residual analysis. 

4. Interact with the user to allow flexible data base 
sub-setting on variables, values of variables and cases; 
all subsets or files created in this way are saved until 
deletion is requested ; 

5, Create undefined data when transformations are undefined; 

6, Allow subfiles to be recombined by intersection or union; 

7. CoLipute, save and display on command relevant summary 
statistics (e.g., mean, max, min, etc.) 

Maintain PMC Interface . Although prompt and efficient data flow is 
the primary responsibility of the DMC, the ASC — charged with ultimate 
responsibility for EEVU evaluation — must fully participate in the cer- 
tification procedures. These procedures involve examination of data at 
the school level (or initiation of "dummy" data packages through the 
system) and comparison with data transmitted from the DMC to tiie ASC. 
These comparisons will be carried out with random frequency and scope, 
toward 

o insuring that data validity is maintained at the highest 

reasonable level; 
o determination of new procedures, organizations, or 

responsibilities that. are necessary for more effective data 

transfer. 

These random samplings will be carried out in an unobtrusive manner 
by the ASC individuals responsible for interaction with schools. 

Alternative Plan for Assigning ISC Role to ASC . At this writing, it 
remains possible that ASC will be asked to assume in part the role of the 
ISC, Should this occur, we would establish an office at each demonstration 
site, and assign to each a full-time employee charged with collection of 
the pertinent data. In substance, this- eventuality would not alter the 
tasks described above, but would require the performance of two additional 
tasks : 
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o ISC establishment and implementation (pre-demonstration) 
o ISC operation (demonstration) 

ISC Establishment and Implementation 

As the role of the ISC is described in the RFP, it will serve as the 
collection (auci alternatively, preparation) agent for the EEVD. As such, 
it must maintain close contact with school, con\inunity, and state officials 
and data sources. It is our belief that the ISC function will require 
one full-time employee per demonstration site. The primary subtasks thus 
af5Sociated with establishment of t;he ISC will include acquisition and 
training of this individual, and specification — for his continued use — 
of collection mechanisms, contacts and techniques. This specification 
will take the form of an ASC-DMC-pr epared guidebook of procedures and 
actions required of the ASC, including specification of collection cycles 
and schedules. 

ISC Operation . Ongoing operation of the ISC will involve largely 
routine performance of the guidebook — specified tasks, and a great deal 
of leg work in expediting data flow and maintaining close interfaces with 
community and educational data sources. 

Confidentiality of Data 

Confidentiality of data as used here refers to two issues: (1) as- 
suring that ASC and other contractors release data to the public only as 
agreed by OEO and other cognizant agencies, if any; (2) assurlrkg that 
data provided to researchers in confidence is kept confidential. 

The first issue is normally handled by contractual agreements between 
the research sponsor and contractor, supplemented and altered as necessary 
by formal and informal agreement. In the case of EEVD, this practice 
would presumably be followed in OEO contracts with ASC and DMC. In addi- 
tion, for certain kinds of data, supplementary agreements vrLth EVA might 
be desirable. 

The second issue, assuring the confidentiality and privacy of data 
sources, can create extremely difficult problems. In some cases, the 
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subpoena power has been used to require researchers to provide informa- 
tion about people's history, opinions, etc, that was originally trans- 
mitted to them in confidence. Since the researcher — and in some cases 
his sponsor — normally assures respondents that confidential information 
will not be divulged, major ethical and legal problems may arise when 
such information is divulged for whatever reason — careless talk by 
people with access to data<,t ppor communication of safeguard rules within 
the research project, theft of files, or legal action to mal^.": the files 
available to investigative bodies and parties to court i'^ctiOi.., etc, 

Ratld's general response to this problem in the case of EEVD will 
be to assure that no link exists between confidential information re- 
ceived and the identity of the people it refers to. In practice, this 
means that we will not conduct panel sampling in our survey work. In- 
stead, we will sample de^ novo for each survey, and destroy names and 
address lists after each survey is administered. Thus not even the 
clerks who code survey information will know who the information refers 
to. Nor will interviewers or survey supervisors be able to relate 
survey information to specific sources after the fact. 

This approach naturally involves some costs. If one can maintain 
names associated with data, it is usually possible to check a number of 
interactions among variables — for example, the relations between parent 
attitudes and children's achievement in school. However, in the case of 
EEVD, the benefits to be gained from identifying individuals do not seem 
worth the risks of possible disclosure, or the elaborate safeguards in- 
volved in so-called "link systems," 

Relations with Other Agencies 

The preceding discussion refers to the transmission and protection 
of data acquir-t'd through direct collection or thuough other agencies. In 
practice, if an organization is conducting a large scale study involving 



Alexand^ar W, Astin and Robert F, Boruch, "A Link System for Assur- 
ing Confidentiality of Research Data in Longitudinal Studies," Review of 
Educational Research , November 1970, pp. 615-624, 
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data collection, processing, analysis and dissemination, relations with 
other agencies will tend to revolve around information access issues. 
This Is likely to be the pattern for agencies participating in EEVD. In 
view of the controversial nature of educational vouchers, and the conse- 
quent uncertainties in planning and administration, some special aspects 
are likely to arise, affecting the information flow. 

For example, OEO is unlikely to find a school district that is will- 
ing to implement EEVD exactly along the lines proposed by the Center for 
the Study of Public Policy or by the OEO Requests for Proposal in connec- 
tion with EEVD. Therefore, arrangements with each district participating 
in EEVD are likely to be negotiated. Each district will make its partici- 
pation in EEVD contingent on certain conditions, some of which are likely 
to be inconsistent with the OEO design for EEVD. For example, local 
educational agencies may be unwilling to allow the free flow of informa- 
tion from EVA to ASC, or OEO. Or in the event that OEO does negotiate 
agreement onLthese issues with local authorities, EVA may subsequently 
decide that the political situation does not permit the free flow of 
information previously agreed on. Or EVA may fidd that the pressure of 
events does not allow the establishment of an effective ISC. In any of 
these events, the functions of ASC would be seriously affected. In the 
first case, there might be no real role for ASC. In the second case, 
OEO would have to balance the merits of dropping its support of EEVD 
against the gains from allowing the project to continue on terms that 
would limit the information flow — terms that might drastically restrict 
the rolesoof DMC and ASC. In the third case, the roles of ASC and DMC 
might have to be substantially expanded in order to fill in for the ab- 
sence of an effective ISC operation. 

Therefore, Rand expects the exact definition of the ASC role and 
relations with other agencies to vary accordingt to the situation at each 
demonstration site. It is not possible to foresee the precise set of 
roles and relationships in advance. However, Figure V-5 shows a tenta- 
tive arrangement, previously discussed with OEO, under which EVA/ISC 
would be the primary data source for ASC, with the intermediary of DMC. 
This arrangement would allow for independent ASC access through the 
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survey subcontractor, and direct ASC dealings with the schools (classroom 
observations, interviews, etc.)* 

Under an arrangement of this kind. Rand would plan the following 
liaison arrangements: 

OEO — Project director and deputy (John Pincus, George Hall) 

Survey Subcontractor — Senior staff member (Barbara Williams) 
and at field level, site director 

Data Management Contractor — Senior staff member (John Farquhar) 

Information Systems Contractor — Senior staff member (Barbara 
Williams and Milbrey McLaughlin) and site director 

Educational Voucher Agencjy — Site director 

Technical Assistance Agency — Site director 

Voucher Schools — Senior staff member (Marjorie Rapp) and site 
director 

Local Government Agencies — Deputy project director (Daniel Weiler) 
and site director 

One special set of relationships with local agencies merits partic- 
ular discussion — the relationship between formative and summative 
evaluation. The ASC task is primarily summative in that it is largely 
designed to report on outcomes. But EVA at each site will need to con- 
duct, directly or under contract, formative evaluations aimed at: (1) 
evaluating alternative program goals and methods; (2) defining appropriate 
degree of parental choice and control; (3) evaluating EVA operation, in- 
cluding relations with schools and parents. These elements are a neces- 
sary part of the formulation and progressive revision of EEVD at the 
local level. 

These formative evaluations and the issues that arise during their 
conduct are of considerable interest to ASC. Therefore, ASC should make 
arrangements to receive, through EVA, copies of the documentation pro- 
duced by the local evaluation effort. It may be expected that ASC will 



If EVA carries out ISC role directly, then ISC liaison arrangements 
^ ^Tould apply to EVA. • . 
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be interested in having the local evaluator perform special tasks or adapt 
its methodology to ASC perceptions of research design. Despite the attrac- 
tiveness to ASC of using local evaluators as an additional resource, Rand 
would attempt to avoid excessive content and influence over EVA-spoasored 
evaluations. In a sense, the local evaluations are part of the experi- 
ment anii the ASC should not atteinpt to affect them. Furthermore, it is 
important for ASC to remain unobtrusive and avoid being identified with 
any particular view or group. 

The impc'iftance of unobtrusiveness applies to ASC dealings with all 
EEVD agencies and participants. The ASC siiould rely primarily on public 
data and ether participating agencies for information sources, fo'Cusing 
its direct information-gathering activities on those aspects where It is 
impractical to use intermediarie/s. 

A tentative definition of information ciiannels might be as follov^^s, 
ai\ previously discusised with OEO. 

1. EVA - responsible for the EEVD locally, focus of information 
flows about schools 

2. ISC - reports to EVA and ASC /PMC . 

3. TAA - reports to EVA and OEO. 

4. DMC - provided data by iSC (et al.) as required by ASC (EVA). 

5. ASC - a. Works with ISC to specify data needs 

b. Establishes in consultation witii OEO hypothesis 
to be tested. 

Conducts witii EVA approval, observations, etc, in 
school and community • 

d. With subcontractor conducts survey of community, 
parents, etc. 

e. Specifies cognitive test batteries and noncognitive 
measures for EVA. 

f . Avoids involvement at district levels prematurely . 

g. Makes sure that data flow system from ISC and DM€ 
meets ASC needs, through intermittent verification 
of data. 

h. Supplements data from ISC/DMC as required for 
evaluations. 
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REPQRTS 

The proposed reporting schedule Is shown in Table V-1, abave. The 
reporting system, in addition to providing summary survey and interview 
data as required by OEO, will be designed to cover the basic elements 
of the evaluation as discussed in Sections III and IV. The first major 
report at the end of the pre-demonstration period will Include: 

o Analysis of baseline and pre-demonstration surveys 

o Analysis of historical and baseline data — educational, 

political/social, and economic 
o I>etaileJ plan for EEVD evaluation procedure including 
hypotheses, data sources, and analytical methods 

Annual reports on EEVD during the demonstration period will include: 

o Analysis of the parent/community surveys and their relation 

to pre-demonstration surveys 
G Analysis of educational, political/social , and economic 

effects 

o Analysis of policy implications 

o Proposed rcivisions of analysis ple^n in light of experience 

Tl)e formal quarterly and annual reporting system is only one part of 
the proposed system of communications with OEO. Rand would expect to 
maintain regular informal communications with OEO, adjusting the evaluation 
design and schedule as mutually agreed, and keeping OEO currently informed 
of the progress of the work, notably the results of surveys, interview?, 
and evaluation analyses that might have a particular bearing on the 
conduct of EEVD. 

As noted above, Rand reports would not be disseminated to other 
agencies and the public except under the terms of agrieements with OEO. 
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RAND FACILITIES 

This section provides iaformation about Rand's physical plant and equip- 
ment, notably the computer facility, as previously submitted to OEO. 

Office Facilities 

ITie Rand Corporation owns its office building in Santa Monica, Call Eur- 
nia, comprising 272,000 square feet of space, in which it houses approximately 
1,125 employees. As part of this building. Rand maintains one of the Largest 
special libraries in California; its holdings include some 50,000 books > 
250,000 reports, and 2,700 periodicals. Rand also maintains an office in 
Washington, D.C., and it staffs and administers the New York City-Rand 
Institute. 

Computing Facilities i 

The Rand Computation Center maintains a wide variety of computing 
machinery, programs, and user support. The computer systems available are 
summarized below. In this project, the machine primarily utilized will be 
an IBM 360/65; the other facilities of the Computation Center will be drawn 
upon as needed. 

Systems 

a. IBM 360/65 . The 360/65 is Rand's primary computing system. Pro- 
grams are processed under a monitor system called OS/360, whicli 
offers the following utility programs and programming languages: 



FORMAC 



ALGOL 



ASM 



BASIC 



BIOMED 



FORTRAN 



COBOL 



CSMP 



CPS 



(International Algorithmic Language) 
(Assembler Language) 

(Batch-mode Processor of BASIC — simplified 
algebraic language — programs) 
(Statistical programs from UCLA) 
(Common Business Oriented Language) 
(Conversational Programming System) 
(Continuous Systems Modeling Program) 
(Formula Manipulation Compiler) 
(FORmula TRANslator) 
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GPSS (General Purpose Simulation System) 

ICS (Integrated Graphics System for the S-C 4060 

and for the Video Graphics System) 

MARK IV (File Management System) 

MARVEL (Language for manipulating data in tabular arrays) 

MATLAI^ (System/ 360 Matrix Language) 

MPS (Mathematical Programming System) 

PERSUB (Matrix-Oriented statistical data analysis sub- 
routines 

PL/1 (Programming Language I) 

RPG (Report Program Generator) 
SIMSCRIPT 1.5 
and II 

SORT/MERGE (Data file sorting and merging program) 

SPSS (Statistical Package for the Social Sciences) 

SSP (Scientific Subroutine Package) 

TSP (Time Series Processor) 



IBM 360/20 . The 360/20 is primarily used as a card-processing 
machine, which prints, reproduces, sorts, interprets, and 
collates card decks. 

JOSS (PDP-6) . JOSS is Rand's interactive, time-shared computer 
system designed for small numerical problems. JOSS consoles are 
conn«^cted to the system either over internal Rand telephone lines 
from special office plugs, cr remotely by standard data- communica- 
tions equipment. Some remote use is by authorized teletypes. The 
syf^tera resides in a Digital Equipment Corporation PDP-6 computer. 

S-C 4060 . The Stromberg DatagrphiX (Stromberg-Carlson) 4060 
Stored Program Recording System translates digital data into 
alphanumeric and graphic data, and records the results oa micro- 
film and, optionally, on paper. 

CDC 6600 . This computer is located at Aerospace Corporation and 
is available for use by Rand. It is appropriate for (1) 6600 
programs obtained outside Rand which require prohibitive efforts 
to convert to the 360/65, and (2) problems requiring the special 
accuracy provided by the 60-bit word length of the 6600. 
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Interactlve Systems on the 3fO/65 

a. System Name: Video Graphics . Description: The Rand Video Graphics 
System consists of a number of low-cost personal, TV-like graphic 
terminals within the Rand building, and their associated software. 
The system provides acces4\ to a range of computers, with a high 
level of interaction betweer; the user and his program. Such 
Interaction includes writing, editing, and compiling programs; 
testing and debugging programs; observing programs during execu- 
tion; communicating with the batch-processing system; and preparing 
S-C 4060 output. 

The Rand Video Graphics Project has developed software to exploit 
the capabilities of inexpensive graphic hardware for a wide range 
of programmers and non-programmers. It is expected that such 
graphic terminals will eventually be the principal means of com- 
munication with Rand's computers. 

b. System Name: CPS . CPS is the Conversational Programming System 
distributed as a Type III program by IBM. CPS, a time-sharing 
system cocLHlning many of the features of JOSS (incremental com- 
piler, line editing) with the language and power of System/ 360, 
consists of typewriter terminals, an incremental PL/I subset 
compiler , and facilities for creating files , editing, and sub- 
mitting jobp into the batch system. 

CPS has been installed on the 360/65 with access via typewriter 
and Video Graphic terminals. Rand has modified CPS to provide 
full graphics capabilities including the display of graphics on 
a Video Graphics terminal and the input of data through data 
tablets and light pens. 

c. System Name: BIOMOD . Descriptioa: BIOMOD is an operational 
system designed to enable unsophisticated computer users to 
study models of biological and other dynamic systems through 
model construction and simulation. It operates on the 360/65 
via a Video Graphics console that includes a data Tablet. A 
user constructs a model by drawing block diagrams and hand- 
printing or typing test while receiving immediate feedback 
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about the interpretation of his actions. Each component of a model 
block diagram may be defined either by another block diagram, or 
by one of the other user-oriented languages: analog-computer-like 
elements, algebraic, differential or chemical equations, or Fortran 
statements. During model simulation, displayed curves are continually 
and automatically updated; the user may stop the simulation and plot 
different variables, change scales and/or parameter values, and 
then continue the simulation. 

System Name; Data Analysis System . Description: The data analysis 
system is designed to aid a researcher in accessing his data and to 
assist him in interactively aj^plying an array of analytic procedures 
from a Video Graphic terminal. The system may be used to review raw 
data in tabular or graphical format; restructure a data base by sub- 
setting on cases, variables or data values; and apply standard 
statistical models for hypothesis testing and formulation. Because 
the system provides on-line access to a data base and many steps 
in an analysis proceed at interactive speeds, a researchej/ is able 
to intijnately explore his data to a depth which has previously been 
impractical . 
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Appendix A 

EVALUATION OBJECTIVES AND METHODS: A BIBLIOGRAPHICAL NOTE 

In the course of preparing the EEVD evaluation plan, we selectively 
consulted the literature on the evaluation of social action programs. 
Tliere were two problems that we sought to clarify. The first was the 
objectives of the evaluation. Tlie second was the adoption of an appro- 
priate approach to the research design. In formulating our evaluation 
plan, we utilized this literature as a basis for assessing the relevance 
and validity ox our approach. 

Tlie appended bibliography is a selective list of the materials con- 
sulted. Two compendiums of readings (Caro 1971 2nd Weiss 1971) on evalua- 
tion research are now available, both of which contain overview introduc- 
tions reviewing the "state of the art," plus bibliographies. 

Tlie books by Williams (1971a) and Rivlin (19 71) provide a historical 
perspective on the requiraments of the kind of evaluation research which 
is useful in assessing alternative policy considerations. Williams (1971b) 
addresses the problems which must be confronted in conducting large-scale 
evaluation projects* 

The methodological issues can be most readily identified by a careful 
reading of Weiss and Rein (1970) and Campbell's comment (1970). Cohen 
(1970) has elaborated on Weiss and Rein in his assessment of evaluations 
of education programs. (Also see Campbell 1971, and 1969.) 
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Appendix B 
A FURTHER NOTE ON SURVEYS 

Social survey research is probably the most common, although not the 
most standarclizGd, form of data collection foi acquiring reliable gener- 
alizations about the knowledge, beliefs, preferences, intentions, and 
experiences of specified subsets of the general population- Standardized 
items and scales do exist in survey literature, but they are limited to 
certain subsets of the population and particular research interests. 
Surveys in large scale social demonstrations typically have more complex 
data requirements than can be served by existing standardized items, and 
must therefore contain items suited to new populations and measurement 
requirements. 

The validity and reliability of survey data depend primarily on the 
quality of: (1) conceptualization of the research problem, (2) survey 
itetas, (3) sampling design, (4) techniques of administering the survey 
instrument, and (5) techniques for analyzing and interpreting the survey 
results. We have discussed items 1, 3j 4, and 5 in the body of the text 
and in the report. Survey Research Specifications and Bfe>seline Survey 
Instrument (Field Research Corporation, February 2, 1972) previously 
submitted to GEO. Here we want to present some of the considerations 
that shaped the development of the survey items and summarize the results 
of the pretest of the draft survey instrument. 

DEVELOPMENT OF THE SURVEY INSTRUMENT 

Surveys that must collect complex data on a wide range of items are 
always difficult, particularly among poor people. Word meanings, con- 
ceptual styles, the sense of what is Important, the sense of what is 
private vary considerably between income groups, and within income groups 
over time; historical events in the community can create new sensitivities 
and new ways of conceptualizing old ideas in a respondent population. 
Middle income people most frequently devise the survey instruments which 
are applied to lower income people, so that disparities of interpretation 
between questions and answers are almost inevitable. This particular 
problem has been frequently discussed in survey literature and does not 
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need explication here. \^at is important are the ways in which we have 
tried to make questions meaningful to respondents and their answers 
meaningful to us. 

Past experience is one of the first aids to the construction of new 
survey instruments and we did make use of instruments which had been used 
in other program evaluations, together with tlie conventional wisdom that 
has developed around interviewing poor people. However, conventional 
wisdom is less rich for chicano respondents than for black and white low 
income populations* Experience suggests Chat chicano respondents yield 
disproportionate numbers of "don't know" answers and answers which seem 
intended primarily to satisfy the intervi ewer. Though part of the reason 
is certainly a language problem diminished by careful translation of the 
survey instrument into Spanish, the problem remains even when local people 
conduct the interviews in Spanish. Our best insight rrom discussions with 
people knowledgeable about chicano communities was that direct answers to 
a structured set of questions are, in form, alien to the interaction and 
conceptual style of Spanish speaking cultures. That suggests the most 
appropriate interviewing style in chicano communities would be a very 
unstructured, indirect conversational flow between interviewer and 
respondent ... a style impossible to use in a study of this type. We 
have compromised by including more open-ended questions than we originally 
intended and have begun the interview with questions about children, which 
is one of the most interesting things for parents to talk about. 

Our first pretest was done with about 20 respondents; the majority 
were chicanos from low income ^reas. Its results were used to eliminate 
items that were intolerable to the respondents, to discover flaws in the 
ordering and wording of questions, and to test the length of the instru- 
ment. The major results of that pretest are reported below. We intend 
to do a second pretest of the instrument early in the eighteen month 
planning period. It will be done with 75 to 100 respondents and with 
purposes: (1) to test the translations of the instruraent into Spanish, 
(2) to continue to smooth the instrument, (3) to develop codes which can 
be used by interviewers in many of the open-ended questions, (4) to 
eliminate items that are too insensitive to pick up distributions. 
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RESULTS OF FIRST PRETEST 

The average length of the interview was over two hours. This obvi- 
ously must be cut, although it was heartening to note that the quality 
of response did not seem to suffer adversely from respondent fatigue. 
For example, some of the best responses to the open-ends came on pages 
17 and 18 of the survey form — more than midway through the interview. 
Though interviewers were distressed by the length of the instrument, 
they found it interesting to administer and sufficiently varied to hold 
respondent interest. We plan to reduce thC; baseline instrument to an 
hour and a half by the question eliminabion and preceding that will 
result from the second pretest. We have already elimiated a long sei'ies 
of "personal control" items because the respondents could not relate to 
the wording and refused to answer them. 

The flow of- the instrument was satisfactory. The series on children 
and the series in which- respondents rated schools on a ladder from best 
to worfjt were especially popular. This is quite interesting in that 
respondents were not able to relate to questions which asked, "Which is 
the best school; which is the worst school"; they could place them rela- 
tive to one another, however. 

Many of the questions seemed "wordy" both to the interviewers and 
to the respondents. Tolerance for listening is apparently low among the 
respondents. Most "word" problems, however, seemed to be associated with 
language use. For chicano respondents, translation of the instrument 
into Spanish is the obvious solution. 

Some interesting substantive issues came to our attention during 
the pretest — all verifying the point that final surveys should be con- 
structed in the context of the community in which they will be given and 
close to the time they will be given. 

o There were many special programs and special classes in the 
area of San Jose where the pretest was done, meaning that 
children are bused from one school to another with some 
frequency. This gives parents experience vrith more than 
one school and complicates opinions about particular schools 
and their offerings. 
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o The greatest antagorisms toward schools involved split shifts 
and split classes. 

o We will have to include questions about Spanish-speaking teachers 
in addition to those on the ethnic identity of teachers; for the 
chicanos, language is often more important than ethnic affili- 
ation. Further, some parents thought that ethnic identity 
between students, teachers, and administrators not only "didn't 
help," but actually was a bad th.'^mg; we will have to precode 
that response. 

o Respondents have strong images of public and parochial schools 
and would tolerate more detailed questions than we have about 
them. (It ±s not obvious that we need more detailed information.) 

o The items (volunteered) that would 'cause parents to change their 
children's schools ranged from teacher quality, type of students, 
r/jmber of sLadents, classroom scheduling, lunch program, to 
access to toilets. 

o The answers to the question series on political mobilization 
(Nos, 88-92 in the draft survey instrument) show the usefulness 
of interviewing both husband and wife where possible. Sometimes 
the wife would not do any of the things but her husband would do 
all of them. 

PRE-DEMONSTRATION AND YEARLY SURVEY INSTRUMENTS 

While the baseline survey instrument contains the majority of items 
that will appear in succeeding surveys, it should be noted that these 
later surveys will have to include questions relating to the specific 
experience of parents with the EEVD, includi:ag their contracts with the 
EVA and thair use of and experiences with EEVD options. These questions 
should add about 20 minutes to the baseline instrument, making a pre- 
demonstration instrument approximately an hour and fifty minutes in length. 
Survey costs have been estimated on the basis of a 110 minur^ii interview, 
but we will make every effort Go reduce the length. Items that can be 
deleted from the pre-demonstration instrument should be indicated in the 
first analysis of the baseline survey results, for example. 
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Additions to the survey Instruments may also be necessary from Mme 
to time. These instruments must be flexible enough to include modifica- 
tions required by unanticipated evenets as the demonstration proceeds. 
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Appendix C 

PROGRAM AND RESOURCE ANALYSIS IN EDUCATIONAL PLANNING "^ 
INTRODUCTION 

Any examination of alternative educational programs must be con- 
cerned with their effectiveness and cost. Because student performance 
is one of the measures of the ef fecUiveneas of the program, a great 
deal of attention is being given to the problems of setting criteria 
of achievement and measuring educational outcome. Less attention has 
been paid to the equally demanding task of estimating and analyzing the 
cost of educational programs. If the instructional strategy of new 
programs is to be successfully utilized by educational planners, infor- 
mation about the cost as well as the effectiveness must be available 
to the decisionmaker. 

This paper explores the conceptual and methodological basis of cost 
analysis and develops a planning cost model for estimating program cost 
for use in evaluating alternative programs and in pre-implementation 
planning for future programs. The planning cost model with its support- 
ing cost analysis methodology provides a consistent basis tor esti- 
mating the dollar cost of educational programs. The development of the 
model was undertaken because the current state of the art in costing 
educational programs does not provide a comparable basis for evaluating 
alternative programs. The usual practice is to give the cost per stu- 
dent for a program with no indication of what is included in the cost* 

When the cost per unit of achievement is used, both the cost and 
the effectiveness measurement problems are severe. Kducation Ttnmkey 
News has drawn attention to several aspects of using this ratio: 

This appendix has been published separately as, '^Program {\^st 
Analysis in Educational Planning," Sue A. Haggart, P-4744, December 
1971. 
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Even when accurate coats are obtained, it is difficult to com- 
pare them with school costs to see which Is less, since school 
costs are kept and reported differently. The comparisons may 
reveal nothing more than different figures, especially since 
the firms [performance contractors in the context of this quo- 
tation] may depreciate certain items much more rapidly than 
schools.,.. It is even more difficult to try to contrast ef- 
fectiveness with cost. If effectiveness is reported in tenths 
of a year's achievement, which some statisticians feel is cut- 
ting it too closely, and that figure is divided into cost data 
which is part hidden and part hypothetical, what does the pub- 
lic get? Will a school board really base a major decision on 
curricular changes on such a "cost per unit of achievement" 
figure?* 

The ratios of cost per student and of cost per unit of achievement are 
widely used, probably because of the false confidence the "number" en- 
genders and the relative ease with which it can be generated. In most 
instances, either ratio masquerades as the output of cost-effectiveness 
analysis. Wisely used, cost-effectiveness analysis of educational pro- 
grams produces several outputs — the aspects of cost, the measures of 
effectiveness, and the relationships betioeen cost and effectiveness . 
The problems and the appropriate use of cost-effectiveness analysis in 

educational planning have been discussed in Cost-Effectiveness Analysis 

t 

for. Educational Planning. Only very seldom is a ratio of cost per stu- 
dent or cost per unit achievement the appropriate end result of a cost- 
effectiveness analysis. 

The planning cost model and its supporting methodology of educa- 
tional program cost analysis provide a solid basis for resolving, at 
least in part, the problems encountered in determining the "cost" of 
educational programs. The planning cost model assists in developing 
comparable cost estimates of alternative programs. In this way, the 
model directly addresses the problems inherent in using an undefined 
cost per student in evaluation of different programs. 

In estimating the program cost to be used in comparing programs ^ 
the resources available within a specific district or assets inherited 
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from discontinued programs are not taken into account, and a standard 
price for common resources, such as teachers, is used. The resulting 
estimated program cost is identified as the oompavahle replication cost. 
It is, in essence, a comparable cost that normalizes the cost of programs 

In estimating the program cost to be used in deciding whether or 
not a particular program can be implemented in a specific district, the 
resources available within the district and district-specific prices 
for these resources must both be determined. The resulting estimated 
program cost in this case is the incremental ciost to the district. 

The role of the planning cost model in estimating both the compa- 
rable replication cost and the incremental cost is pictured in Fig. 1. 
In this process, the first step, common to estimating either the compa- 
rable replication cost or the incremental cost, is a definition of the 
program in terms of its objectives, its students, and its resource re- 
quirements. These resource requirements are translated into the type 
of program cost estimate relevant to the decision to be made. The plan- 
ning cost model, by providing a consistent methodology for estimating 
program cost, helps insure cost comparability among programs for deci- 
sionmaking purposes. 

Before describing the planning cost model, a short discussion of 
the concepts and techniques of cost analysis underlying the development 
of the model should be helpful. The use of the model in estimating 
the comparable replication cost and the incremental cost is illustrated 
in the final part. 

COST ANALYSIS 

Cost analysis is concerned with the determination of physical re- 
source requirements for the program, with calculating the program dollar 
cost, and with systematically evaluating the impact of changes in the 
program on both the resources needed and their dollar cost. The ap- 
proach is to first determine the facilities, staff, equipment, materi- 
als, and services needed to conduct the educational program and to then 
translate these resource requirements into an estimated program cost. 
This sequence forces explicit consideration of the varying resource re- 
uirements for different programs or for changes in program scope. 
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The educational program has as its core an instructional strategy. 
This instructional strategy includes both the resources and the way in 
which the resources are used to produce the educational outcome. 

Definition of the Educational Pro/ ^ ram 

The first step in analyzing the resource requirements and cost of 
a program is the definition of the program. The quality of the estimate 
of the cost of an educational program depends on the completeness with 
which the resource requirements of the program are determined. This 
determination, In turn, depends on the description of the educational 
program. The sequence of events then begins with a description of what 
the program is and how the program works and continues with a determina- 
tion cf the quality and quantity of the resources. These resource re- 
quirements are translated into an estimate of the program dollar cost. 
In defining the program* the types and magnitude of support activities 
or services also need to be identified. 

Determination of Resource Requirenien ts 

The definition of the educational program is followed by the deter- 
mination of the resource requirements. The data required are arrayed 
in the illustrative format of Fig. 2. Some of the categories in Fig. 2 
pertain to resources directly. Others are 'Afunctional packages/* such 
as training, which are combinations of resource items. Additional data 
should be provided as appropriate for specific programs,. Each of the 
Items in the format will be defined in terms of the kind of information 
needeJ. 

Data about the characteristics of the students served and the num- 
ber of students in the program will, of course, be the same data required 
for the evaluation of the effectiveness of the program. Data on other 
district conditions that might have an effect on the outcome, such as 
income level, turnover rate^ or mobility, should be provided. The in- 
structional time should be given, along with other information that re- 
lates to determining the actual time spent with subgroups of students 
or individual students^ The student-teacher ratio is usually used as 
a proxy for this, but an effort should be made to refine this piece of 
information. 



-162- 



Characteristics of Students Served 

Number of Students 

Instructional Data 
Class time 
Class size 

Facil itic»s 
Space 

Students /classroom/ day 

Utilization 

Furnishings 

Staffing 
Teachers 

Special teachers 
Paraprof ess lona Is 
Other personnel 

Equipment 

Program-related 
Student-related 

Materials 

Program-related 
Student-related 

Pre-service Training 

In-service Training 

Other Support 



Fig. 2 — Format fov ppogram and resource infonnation 



In describing the facilities needed, the space requirements, in- 
cluding mobile or portable classrooms, laboratories, and their utiliza- 
tion rates, should be carefully d^temilned. The requirements for non* 
school facilities should also be stated. The special needs for electrical 
outlets, air conditioning, carpeting, and lighting should be identified. 
Furniture needs are to be specified, identifying any special per-student 
requirements. 
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Staffing for the proRram should he doscribcd in terms of thr quali- 
fications needed as well as In terms of nuinhor (e.^., Rive numher of 
rertlflcated or certified teachers, the number of special tearhers, 
paraprof esslonal staff, and other personnel Involved in tlie program). 
If a staff member works Less t:han ful I time, tlie percent of time In- 
volved should be given. Staff requirements for time beyond the '^normal" 
school day should be stated. This includes, for example, custodial or 
security services needed to keep the school open after the regular day. 

Equipment and materials should be Identified as program-related, 
classroom-related, or student-related. Program-related equipment or 
material Is that which will he used by several students during the day 
or some time period of the program. Very often the equipment or mate- 
rials may be grouped by classroom unit. Student-related equipment or 
material is that which is required because there Is a specific number 
of students in the program. An additional distinction should be made 
about the consummable nature of the materials and about the lifetime 
of. the equipment. The same treatment should be applied to supplies if 
the usual district practice is to treat equipment and supplies as sep- 
arate categories. 

The amount of time involved in pre-servlce and In-service train- 
ing should be specified. The materials or equipment required should 
be given. It should be noted if the training time is Included as part 
of the regular time of the staff or if it is Incrementnl to the regular 
working hours. If in-service training time Is a substantial part of the 
individual teacher^s time, additional teachers (or substitute teachers) 
may be required for the instructional load of the program. 

The requirement for program-related services such as evaluation 
or other management activities should be given. It is preferable if 
thfe actual time or the numbers of consultants can be specified. In 
either case, the purpose is to provide some estimate of the magnitude 
of these services so that the decision can he made on what it costs to 
buy the service rather than to develop, if possible, an In-house capability* 

Support from other activities means the support required by the 
educational program from such service functions as transportation. For 
example, a particular educational program might need bus transportation 
for field trips. This instructipnally-required transportation is over 
and above the cost of home-to-school transportation. 
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The resource requirements Identified In Fig. 2 are meant to be sug- 
gestive only. If other data are available, they should be given, since 
the purpose is to define as completely as possible those resources and 
cost-generating activities needed to carry out the educational program. 

The resource requirements are then translated into the dollar esti- 
mates of program cost — either the comparable replication cost or the 
incremental cost. A planning cost model provides a framework for sys- 
tematically and consistently estimating program cost. 

THE PLANNING COST MODEL 

The planning cost model provides the mechanism to determine, con- 
veniently and consistently, the cost of various alternative programs. 
By design, the model is appropriate for pencil-and-paper operation as 
well as computer operation. 

The model provides the framework for bringing together the resources 
(facilities, staff, equipment, materials) required to carry out an edti- 
cational program and for relating these resources to program output in 
the form of activities. 

By relating the inputs required to produce outputs, in terms of 
activities, the model provides more information for making decisions 
about the merit of selected changes in the activity structure of the 
total program. For example, trade-offs between fewer but longer instruc- 
tional periods and more but shorter periods could be assessed. The 
model also provides the basis for examining the cost consequences, for 
the total program, of changes in the resource utilization rate (i.e*, 
student/ teacher ratio) or in resource cost (i.e., teacher salary). 

The task of constructing the model demanded a close examination 
of the concepts of cost analysis, especially in their application to 
educational program cost methodology. This examination resulted in the 
delineation of an approach to costing educational programs. Basic to 
this is the definition of a preliminary list of cost categories. Those 
costs of school district operation not affected by the existence of the 



A planning cost model designed for computer operation is described 
in R-672-SJS, Pr^ojeat R-3j San Joae^ California: Evaluation of Results 
and Development of a Coat Model, M. L. Rapp, M. B. Carpenter, S. A. Haggart, 
O „ S. H. Landa, and G. C. Sumner, The Rand Corporation, March 1971. 
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program are not Included in t)ie estimated cost of the program. An ex- 
ample will serve to clarify this point. 

The district cost category, transportation, provides for the trans- 
portation of students to and from school. Students In the special pro- 
gram will continue to receive transportation, if they need it, just as 
though they were not in the special program but were, instead, students 
in the regular program. This regular transportation cost is not included 
in the cost of the individual program. But, if the instructional method 
of the special program calls for field (rips or other activities requir- 
ing transportation, the cost of this transportation is included as a 
cost of the special program. 

Cost Categories 

The items, services, people, and activities and their cost required 
for an educational program can be brought together in one format — the 
cost element structure shown in Fig. 3. These cost elements are grouped 
into two broad categories: the acquisition cost and the operational 
cost. The cost of most programs can be adequately encompassed within 

Acquisition Cost Opemtionat Cost 

Design of program ^ Program direction 

Development of materials Evaluation* ^ 

Evaluation design* Management support 

Program implementation Salaries 
Equipment Teachers 

Program-related Paraprof essionals 

Student-related Specialists 
Materials and supplies Other 

Program-related In-service training 

Student-related Materials and siipplies 

Pre-servlce training Program-related 
Facilities (space) Student-related 
Installation Equipment 

Replacement 
Maintenance 
Facilities O&M 
Contracted services 
Media services 
Transportation 



In an operational program, ns opposed to a demonstration 
program, there might be no program cost associated with these 
activities . 



Fig. 3 — Coat elamr>ni stvuoture for 'ed}{aational programs 



-166- 



these two bro«id categories. The ncqtifsltlon cost Is the one-time lost 
to acquire n capability. The <»perat lon.nl cost Is the continulnR cost 
to maintain a capability over a period of time. In the following dis- 
cussion, one year's operating cost Is assumed. 

The acquisition, or one-time, cost to acquire a capability Is, In 
practice, also referred to as Initial, investment, or capital cost. It 
covers the cost of all resources required to acquire a capability. Tlic* 
cost of the effort devoted to research, development, or design of com- 
ponents of the program or alternatives should be Included as part of 
this cost. The cost of designing a different mathematics curriculum, 
for example, is a development cost. In estimating the (^ompamble vcp^ 
liaation cost, however, some overall development costs might be treated 
as sunk costs. That Is, the first program to use the new curriculum 
would Incur this expense, and subsequent programs using the curriculun 
would Inherit the new curriculum on a cost-free basis. On th' other 
hand. If the curriculum had to be redesigned for a patticular program, 
this would be a development cost for that program. 

The operational cost is also referred to as tlie recurring or con- 
tinuing cost to maintain the capability. The cost of modification of 
facilities and the cost of in-service training of teachers are included 
as an operational cost to maintain the program. These broad categories 
of cost — acquisition and operational — are used as a basis for organizing 
the cost elements into the cost element structure. 

This structure provides the framework for identifying the cost of 
the program in an operational environment. Each element, whether it 
is an item purchased or an estimate of activity cost, will be discussed. 
But first, remember that costs not varying because of the existence of 
the program are not included. For example, district-wide administrative 
costs are not allocated. 

Costs that might be incurred in a demonstration program but not in 
an opevational program are identified by an asterisk in Fig. 3. Some 
of the cost categories can be characterized as the cost of activities 
rather than the cost of items purchased. In many Instances, the items 
purchased quite clearly underlie the cost of activities, but the activ- 
ity cost, however, may be used directly in estimating the program cost. 
For example, the evaluation cost of a program might be estimated by 
48lng a factor such as cost per student. Or, the cost per program might 
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be used if the evaluation is done by an outside contractor or evaluator. 
If appropriate, these would be the factors used to estimate the opera- 
tional cost of evaluation. The acquisition cost — the non-recurring 
cost — for evaluation might be based on the district staff time to design 
the evaluation of the program or might simply be the cof>t charged by the 
outside evaluator. The cost basis for these inputs would be per pvogvam 
for acquisition cost and per student or program for the operational cost. 

Cost Basis for Inputs 

The cost basis for all inputs for the categories in the cost ele- 
ment structure is shown in Table 1. For each category the cost basis 

Table 1 
THE COST BASIS FOR INPUTS 

Cost Basis 



Categories Student Program Unit Service 

Acquisition Cost 

Design of program x 
Development of materials x 
Evaluation dasign x 
Program implementation x 
Equipment 

Program- related x x 

Student-relaned x 
Materials 

Program-related x 

Student-related x 
Pre-service training x x 

Facilities x 

Installation x 

Operational Cost 

Program direction x 
Evaluation x x 

Management support x 
Salaries 

Teachers x 

Paraprof essionals x 

Specialists x x 

Other X X 

In-aervice training x x 

Materials and supplies 

Program-related x x 

Student-related x 
Equipment 

Rp«tlacement x 
Maintens.nce ^ 
Facilities O&M x 
Contracted services x x 

Media services x x 



Transportation x 



X 
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Is either per student, per program, per unit, or direct service charge. 
The per student and per program distinction Is rather obvious; the per 
unit basis refers to units such as classrooms, resource centers, and 
language laboratories. The service basis Is used when the Input to the 
model might be the extent of a service performed either within the dis- 
trict or by an outside source. An example of the fomier would be the 
operation and maintenance of the facilities; the latter service-based 
Input might cover such Items as the contracted transportation for the 
Instructional part of a program or the provision of so many hours of 
Instructional television . 

In some cases, the cost input basis might be a combination of pro- 
gram and unit (classroom), of student and service, or of program and 
service. No rigidity is implied. The intent is to provide an under- 
standing of how the inputs of the model are categorized. This catego- 
rization is basic to the structure of the planning cost model. At this 
time, it is only necessary to emphasize that some level of jnput is re- 
quired because there is a certain number of students, and othei levels 
of input are required because there is a certain number of classrooms 
or instructional centers ^ In many cases, there is a program cost that 
is independent *>i the number of students or centers. 

Outputs and Inputs of the Model 

A program-related cost can be a thruput to the model. For example, 
the cost of program development would be both an input and output. The 
cost of pre-service training for the teachers in the program is calcu- 
lated within the model. The physical descriptors of the program ar.d 
cost factors, such as the number of teachers, the salary cost, the cost 
per mile, are the inputs to the model. The objective is to keep the 
number of inputs to a workable minimum while allowing enough input flex- 
ibility to provide useful outputs of the model for the evaluation and 
planning of educational programs. 

The outputs of the model are, in general, the resource and cost 
information about the specific educational program. The deoariptore 
of the progr(jOn — number of teachers; number of students; rp^ice require- 
ments; equipment, materials, and supplies; and need for services such 
afl transportation or evaluation — are shown right along with the coat. 
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output. The purpose Is to provide. In one place, an estimate of the 
comparable replication cost and a description of what is being bought. 
As this practice becomes more prevalent, the use of a cost per student 
tc describe an unknovm quantity will decrease and the quality of infor^ 
mation available to the educational planner will increase. 

The output of the model is illustrated in Figs. 4 and 5. Notice 
the similarity of the format to the cost element structure of Fig. 3. 
More detailed information for any of the items shown can be provided 
in supporting reports. For example, the resources and cost underlying 
the cost per student hour under Media Services might be of interest for 
some types of decisions. The supporting detail for this would follow 
the same cost element structure used for estimating the cost of the en- 
tire educational program. 



Des c ription of Program 



Program: 
Staffing: 



Objective: 

Student Characteristics; 



Facilities: 
Equipment : 

Materials : 

Acquisition Cost 

Program activities 
Equipment 
Facilities 
Materials 

Total acquisition cost 

Operational Cost 

Program activities 

Salaries 

Materials 

Supplies 

Equipment 

Other supp.^rt 

Total operational cost 



Operational Characterisnics : 
Instructional time 
Student grouping 
Location 



XXX 
XX 
XX 
XX 



$ xxxx 



$ XXX 

xxxx 

XX 
XX 
XX 
XX 

$xxxx« 



Fig. 4 — Swrmry output of the model 
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Acqulsltion Cost 
Program Activities: 



Equipment: 

Facilities: 
Materials : 



Operational Cost 
Program Activities: 



Salaries: 



Design of Program 
Development of Materials 
Evaluation Design 
Program Implementation 
Pre-service Training 
Installation 



Program-related 
Student- related 



Student-related 

Program- related 
Student-related 

Total Acquisition Cost 



Program Direction 
Evaluation 
Management Support 
In-Service Training 
Facilities O&M 
Contracted Services 
Media Services 
Transportation 

Teachers 
Specialists 
Paraprof essionals 
Other 



$ XXX 
XXX 
XXX 

xxxx 

XXX 
XXX 

$ XXX 
XXX 

$ XXX 

$ XX 
XX 



XXX 
XXX 
XX 
XX 
XX 
XX 
XX 
XX 



XXXX 
XXX 
XXX 
XXX 



$xxxxx 



l^xxxxx 



$ xxxx 



Materials: 

Supplies; 

Equip nent: 
Other support; 



Program 
Student 



Program 
Student 

Replacement 
Maintenance 



XX 
XX 



XX 
XX 



XX 
XX 



Total Operational Cost 
Fig. S~ Detailed output of program oast estimate 



XXX 



XX 



XX 
XX 



$xxxxx 



ERIC 
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Th<^ inputs of the model fall into three broad groups: (1) the 
physical descriptors of the program; (2) the cost of resoiirres and ser- 
vices; and (3) the factors or estimating relationships. The physical 
descriptors, including the type and quantity of resources, were shown- 
in Fig. 2, Format for Program and Resource Information. In short, these 
inputs describe the students, the educational program, and the resource 
requirements. Inputs are required for all the changes, or variables, 
that make one program different from another program. 

The inputs describe the cost of resources and services and cover 
such items as the cost of equipment used, the salaries of the staff, 
the cost of testing, the cost of transportation, and the cost of train- 
ing. The input factors, or estimating relationships, include both cost 
factors such as cost of materials per student and non-cost estimating 
relationships such as number of in-service training days per teacher. 

The Structure of the Model 

The model integrates the program description, in terms of resources 
required, with the process of estimating the program cost. This process 
begins with the determination of resource requirements and continues * 
with the translation of these resource requirements into an estimate 
of dollar cost. Both the acquisition cost and trie operational cost are 
estimated • 

The model's framework for estimating the acquisition and the op- 
erational cost is shown in Figs. 6 and 7, respectively. For each cost 
category there is an estimate of cost on either a student, program, unit, 
or service basis. In the case of ''units," the estimate can be the cost 
per teacher, the cost of the equipment per classroom or instructional 
center, or the cost per student or materials consumed. For some cost 
categories, the estimate can be based on an overall -program cost. For 
example, the pre-service training, if done by an oiitside contractor, 
might be a total cost for the program. It could also be a cost per teacher. 

In the cost category for Materials, the cost estimate may require 
an estimate for the cost for student-related materials, for the cost 
of materials in the classroom for use by many students, and for the cost 
of program materials used by the staff in conducting the program* The 
same practice is followed for the cost categories of the framework for 




".e operational cost in Fig. 7. 
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The cost categories provide a convenient way to identify the data 
needed about the educational proj;rani and its operation in order to e55- 
timate its cost. The data * or the cost cate^^ories for both the acqui- 
sition and operational cost are shown separately in Fip^s, 8 and 9, 
respectively, 

USE OF THE PIJiNNING COST MODEL 

Estimating the Comparable Replication Cost 

The use of the model will be illustrated by estimating the compa- 
rable replication cost for several different programs. It should be 
emphasized that in order to compare programs in different districts, 
comparable resources prices and salaries have to be used, A comparison 
of actual costs would have little meaning since the differences among 
programs would not only reflect differences in the programs but also 
differences in teacher salaries and other local prices. 

As shown in Fig, 1, the process of estimating the comparable rep- 
lication cost and the incremental cost for a program begins with a de- 
scription of the program and its resource requirements. This informa- 
tion is then processed through the model in order to estimate the cost. 
The description of the program includes both program information and 
resource information as shv,«^n in the format of Fig, 2, 

The program and resource data I'^or several illustrative programs 
are given in detail in the appendix, The summary of this information 
is given in Table 2, The resource requirements are estimates of what 
it would take to repllcaf:e the Instructional strategy of the program. 

The information under Other Support provides an example. In the 
replicated program, there Is an item for consultants to the program. 
It is estimated as approximately eight days for the year of program op- 
eration. This is an estimate of what might be needed in a ixituce pro- 
gram rather than nn estimate of what was used In past programs. The 
same is true for Program Evaluation. A category for this type of activ- 
ity calls attention to the need for evaluation of the program even in 
operrtlon as part of the regular district programs. In the estimate 
for Che replication cost, this category Incurs a cost per student for 



O evaiuatlon of the progra^n. 

ERIC 
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Coat Categornf 

Design of Program 
Development of Material 
Evaluation Design 
Program Implementation 
P re-service Training 
Installation 

Equipment 



Facilities 



Materials 



Data Reqidr^ements 

If these activities are required for the 
program, the number ^ the type of personnel 
involved, the time spent, and salary are 
needed. 



The equipTn^nt list is determined for each 
studrnt, for each classroom, and, if appli- 
cable, for the program. Tlie classroom's 
equipment is used by several classes of 
students. The number of students that can 
use the equipment is specified. 

The space required is that over and above 
the regular program; both for each student 
or for special resource centers. 

The initial sto^k of materials is deter- 
mined for each student, for each classroom, 
and, if applicable, for the program. 



Fig, 8 — Program data — acquisition cost categories 



Cost Category 

Program Direction 
Evaluation 
Managenvent Support 

Salaries (with fringe 
benefits) 



Materials and Supplies 



Equipment 



Facilities O&M 
Contracted Services 
Media Sen^'lces 
Trans port^.tion 



Data Requirements 

o The number and type of scsff, the time spent 
for each activity, and salary are needed for 
this . 



All instructional staff and direct support 
classes of staff ate identified by broad 
category; i.e. , general teachers, special- 
ists, and aides rather than a teacher with a 
specific salary are used< Fringe benefits 
are included at the district percentage 
factor. 

The type and quantity of n»aterials usf<d are 
specified on a student and program basis* 

The equipment maintenance factor and the 
equipment replacement factor (based on the 
estimated lifetime of the equipment) are 
applied to the equipment used in the program. 

The program requirements for each of the 
categories ate specified in terms of square 
feet 'maintained, services purchased, nunbar 
of hourc of audio-visual instruction and 
bus trip m.Ileage. 



Fig. 9 — Program data — operational coat categories 
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Table 2 

PROGRAM RESOURCE REQUIREMENTS 



Item 


Program A 


Pi*ajfcvn B 


Prcgvcuri C 


Program 0 


Prcgram B 


Progtxm F 


Number of Students: Reading 


350 


2B5 


491 


150 


103 


250 


Math 


350 


285 • 


535 


150 


103 




Instructional Time: Reading 


1 


1 


1 


1.25^ 


1.25 


1 


(In hours) Math 


1 


1 


1 


1.25 


1.25 




Facilities 














( 

Space ) 


"5 trailers 


4 trailers 


« , b 
2 sgl centers 


1 classroom 


1 classroom 


2 " classrooms 


2 classrooms 


1 classroom 


1 dbl center^ 


1 activity area 


1 artlviiy area 






900/3000 


1600/3000 


1 reinforcement 




• Total square feat 


5600 


4600 


. 8000 


2000 • 


2000 




Air conditioned 


X 




X 


X 




X 


Carpeted 


X 




X 


X 


X 


X 


Spcclsl wiring 


X 


X 


X 


X 




X 


Carr*U 


X 


X 










Tables 


X 


X 


X 


X 






Utilization 














Time In use 3(2-hr)shlf ts 


3(2-hr)shlfts 


7 periods 


5** 


5 


«i 


Student/instructional unit 


20 


20 


*0-S; 65-1) 


50 


50 


25 


Area/student (sq ft}- 


50 


50 


50 


40 


40 


40 


Staffing 








Tcachers/ccntnr or unit 


1 


1 


1 • 


1 


1 




Paraprof esslonals/unlt 


■I 


1 


1 


2 


J 


\ 1 

i 


Students per frachvr 


20 


20 


40/60 


50 


50 


25 


Tecchers per program 


6 


5 


4 


1 • 


1 


"2 


Paraprofcsslonals/ program 


6 


5 


5 


2 


3 




Other direct 














Equipment 














(• 


[)orBett M-66 


EDL AUD-X 


Hoffman readers 


Telex 


Telex 


Cassette 


Kajor Itena \ 


teaching 


Control led 


Tape recorders. 


Cassette 


Cassette 


players 


machine 


readers 


Flashcard rdrs • 


recorders 


recorders 


Tape recorders 






Tach-X 


Borg-Warncr 60 


Tape recorders 


Tape recorders 




Materials 




Flash-X 


(backup) 


Language master 




Program-related . Fllmstrlps 


Fllmstrlps 


Hoffman matls 


BRL materials 


BRL materials 


Fllmstrlps 


Records 


Discs 


EDL materials 


Cassettes 


Cassettes 


Cassettes 


Dorset t 


EDL OAterlala 


Great variety 


Variety of 


Variety of 


Paperbacks 


Consumables 


materials 




Borg-Warncr matls other 


other 












(stu<3ent-related) 


X 


-X 


X 


X 






Pre-servlce training 










X 


X 


Teachers 2 weeks 


1 week 


2 weeks 


1 week 


1 week 


1 week 


Psrsprofesslonals 2 weeks 


1 week 




1 week 


1 week 




Other staff 


... 






1 week 






In^sarvice training 
Other Support 


S days 




2 hr/wk 


4 days 


3 days 


3 days 


Student diagnostic services 








c 


c 




Program evaluation 


X 


X 




X 


X 




Coos ul tan ts 8 dsys 

J 

- a_ 


6 days 


X 

6 days 


X 

8 day a 


X. 

8 days 


X 

8 da/s 



^Two 75-»lnutc periods for grades 1^4 with reinforcement In regular classes 
Each center has an Instructional area plus an activity araa« 
remote diagnostic and prescriptive services » 



One 2,25-hr period grades 5 and 6» 



ERIC 
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The dollar cost Information for these illustrative programs is shown 
shown in Table 3, These are for the estimates of the comparable repli- 
cation cost^ This infonration is combined with the program and resource 
information of Table 2 and provides the basic input information for the 
planning cost models 

The standard input costs and the factors for use in the planning 
cost model are given in Tig. 10 ► The term "standard'* is used as a de- 
scription of the factor used across all programs, 

A cost of $12,000 per year per teacher is used in the model to 
estimate the comparable replication cost. This includes the fringe ben- 
efits (fixed charges in most district accounting systems). This is ob- 
viously out of line for> say. a small rural district in the southeast- 
ern part of the country ► But because this factor was used for all the 
programs, the different cost for the salary expense of the program cost 
actually reflects the difference in the number of teachers needed for 
the program* This same argument applies to all the standard resource 
CO- ts and factors used in the planning cost model. 

The comparable replication cost for each of the illustrative pro- 
grams is given in Table A* The acquisition cost includes the cost to 
remodel and furnish the instructional centers, the cost of the equip- 
ment and the materials needed for all the instructional cente^rs, and 
the pre-service training cost of the program staff. The operational 
cost includes the ^salaries of the staff, the cost of materials consumed 
or lost through attrition or theft, the cost of replacing and maintain- 
ing the equipmsnt, the cost of in-service training, and other support, 
which includes a program evaluation cost on a per-student basis per year 
and consultants required during the year. The comparable replication 
cost along with the relevant dimensions of the specific programs is 
summarized in Table 5» 

The estima'^ion of the comparable replication ccst has an advantage 
In addition to adjusting for variations in the resource prices so that 
the cost of programs in different districts is on a comparable basis. 
This advantage li^tj2 In the discipline neceB»ary to organize the program 
Information and the cost information. In Table 5, the operational cost 
per student per subject offers a quick conparlson of the rel^^^ive merits 
of the programs. The other data rf Table 5 can be analyzed .in a similar 
•!(^ihion. Care must be tak;(5n, hov/cv-ir, not to develop misleading "results. 



t 
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Table 3 
PROGRAM COST INFORMATION 
(Costs in dollars) 



I'^'OQVco^ A ProgVim B Pro-jvcvn C Frogvcon P Ppogrwn K Progra^ri y 



Equipm:!nt Cost 
TotaD 

Cost per instruct iona] art^a 
Niimlicr of instructional arei's 
Students per instruct'! area 
Replacement — 10 percent 
Maintenance — 10/20 percent 
Materials Cost 
Total 

Cor>t per instructional area 

Number of instructional areas 

Consunobles ($ per student) 
Pre-scrvlce Trainius 

Kumlier of staff days^ 

Cost per day^ 

Tot.ll cost 
In-sorvicc Training 

Number of staff-days 

Cost pur day 

Total cost 
Other Support 

Student diagnostic services 

Student evaluation ($/studnnt) 

Consul tanJ K ($100/.li<y) 



20,^00 


15,000 


37,000 


2,500 


2 


,000 


5,000 


3,A00 


3,000 


9,230^ 


2,500 


2 


,000 


2 ,500 


6 


5 




1^ 




1^ 


2 


20 


20 


40/65^ 


50 




50 


25 


2,0/.0 


1 ,500 


3,700 


250 




200 


500 


^,080 


3,000 


7,800 


2 30 




200 


500 


18,;. 00 


20,000 


^5,000 


8,000 


8 


,600 


7,600 


3,000 


A,ono 


n,230''» 


8,000 


8 


,o00 


3,800 


6 


5 




2 - 




2 . 


2 


10 


. 10 


10 


10 




10 


5 


120 


50 


90 


15 




20 


20 


200 


200 


200 


200 




200 


200 


2A,000 


1C,000 


18,000 


3.000 . 


U 


,000 


U ,000 


30 




32 


J2 




12 


12 


200 




200 


200 




200 


200 


6,000 




6,^00 


2,A00 


2 


,^00 


2,^00 








50^ 




50^ 




10 


10 


10 


10 




JO 


10 


800 


800 


800 


BOO 




800 


800 



Cost per center includes reinforcement areas. Single center cost slightly more than cost shown. 

One classroom arei* plus one activity area. 

Forty students per dingle center, sixty-five per double. 

Includes time for paraprof essional training. 

Includes salary, materials, and training costs. 

Remote diagnostic and prescriptive services. 
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Facilltles 

Remodel Ing (Including carpeting , 

alrcondltlonlng , etc.) $ 3 ,()00/center 

Fiirnlslilngs (Including ^arrels) $ 2,000/center 

Eq uipment 

Replacement 10% 

Maintenance (depends on estimate of 

reliability based on complexity) 10% or 20% 

Materials 

Attrition from use, theft 10% 

Consumables $10/studcnt 

Salaries (including fringe benefits) 

Teachers $12,000/year 

Paraprof esslonal s $ 5 ,000 /year 

Specialists $12,000/year 

Program directors $]5,000/year 

General support $10,000/year 

General administrative $12,000/year 

Consultants $100/day 

Pre- and In-eervice Training (including 

salaries , materials , training) $200 /day 

Program Evaluation $10/student 



Fig, 10 — Stanc-jrd veeouvce costs and faotovs 
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Table 4 

COMPARABLE REPLICATION COST FOR THK ILLUSTRATIVE PROGRAMS 

(In dollars) 



Acquisition Cost 

Facilities (rn.rjodel, furnish) 

Total program cost. 

(Cost/tnstructif^naJ area) 
Kqulpnient 

Tot.Tl program cost 

(Cost/ Instruct loun I ar<?a) 
Matcjr J nls 

Total pro grain cost 

(Cost por instructional arn.n) 
Pri^'-sorvf ce Training; 

Total acquif5ition cost 

OpcratioKal Coc^t 

Salaries (Incl frlrp.e benefits) 

Teachers. ($12,nC0/yr) 

Paraprofesslonals ($3,000/yr) 

Other (variable) 
Materials 

Program-related (10%) 

Ccucumablc.s (student) 

EquIpiiiUUL 

Hep 3 a cement 

Maintenance 
In-jiervice Training 
Other Support 

Student diagnostic 

Student evaluation 

Consultants (<:lOO/dny) 
Total operational cost 



serviceij 
(tes tin^) 



Progruri A VvnrjV(.:r B Fx'ogvam C Program D Program E Pronrjn F 



30,000 
(5,000) 


25,000 
(5,000) 


20,000 
(5,000) 


?,500 
(3,750) 


7,500 
(3,750) 


10,000 
(5,UO0) 


20,^^00 
(3,A00) 


15,000 
(3,000) 


?7,000 
(9,250) 


2,500 
(2,500) 


Z,000 
(2,000) 


5 , 000 
(2,500) 


18,000 
(3,000) 
2Aj_000 
92,400 


20,000 
(4,000) 
10,000 
70,000 


A5,000 
(U,230) 

13j000 
1^0,000' 


8,000 
(8.000) 
3^000 

n,ooo 


8,600 
(8,600) 

22,100 


7,000 
(3,H0(1) 

26,^00 


72,000 
30,000 


00,000 
2i>,000 


a8,000 

25,000 


12,000 
10,000 


12,000 
15,000 


?A,00<) 
1 0 , 000 


1,800 
3,500 


2,000 
2,850 


A, 500 
5,oi>0 


80G 
1,SOO 


800 
1,030 


7^}0 
2,500 


2,0A0 
4,080 
6,000 


1,500 
3,000 


3,700 
7,800 
6,A0O 


250 
25r 
2,AU0 


200 
200 
2,A0O 


500 
500 
2^A00 


;j,500 
SOO 
123,720 


2,850 
800 
98,000 


5,000 
800 
106,200 


7.500^ 
1,500 
80O 
37,000 


5,000^ 
1 ,000 
800 
38,A90 


» 

2/5t}0 

rpco 



Remote diagnostic and prescriptive services. 



ERIC 
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For sxample, the acquisition cost per student could be obtained^, ^ 
It seems, simply by dividing the acquisition cost by the number o^ystu- . 
dents. The problem lies in just what "number'* of students to use,^ IF 
the total number of students in all the Instructional periods (or some^ 
such time division) is used, -the acquisition cost per student reflects 
an implicit utllizatinn rate for the instructional center, A case in 
point is Program C, In that program, the Instructional centers are iisod 
seven periods (or hours) each day. In current practice, that is the 
maximum utilization rate f or f aci Lities in any one day. In Program A, 
on the other hand, if the instructional centers had been used for seven 
periods instead of six, one less instructional center would have had 
to be furnished. 

If the number of students per instructional center Is assumed as 
"best," then the acquisition cost on a per^student basis for each in- 
structional center for each program can be obtained and qualified by a- , 
statement of the utilization rate of the instructional centers. An ob- 
stacle is encountered in using the acquisition cost per student per pro- 
gram. That is, that the equipment and materials purchased for one year 
will have more than one year's service as ^he program is continued. In 
short, the use of the acquisition cost per student as an Indicator of. 
program cost is fraught with hazards.'. These hazards' are explored in the 
section on estimating the incremental cost of a specific program in a 
particular district. 

Estimating the Incremental Cost 

The comparable replication cost serves as an "Index" cost for use 
in the comparative analysis of different programs. It does hot answer 
the question of what a new program might cost if Implemented in a spe- 
cific school district. The Inciemental cost to the district is necessary 
in making decisions about whether or not the district can afford a pro- 
gram similar to the successful program In another district. This cost 
is necessary when deciding the scope and the design of the program that 
can be accoiranodated within the resource constraints of the district. 

The process of estimating the Incremental cost is essentially the 
same as the process of estimating the comparable replication cost. The 
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emphasis Is on estimating the resource requirements and on translating, 
these requirements Into an estimate of cost. In some districts, the un- 
ivailablllty of certain resources might be an obstacle to the Impleioen- 
tatlon of a program even though the district had the funds to afford the 
program in an acoounting sense. This possibility makes It all the more 
Important to estimate the physical resources needed to Implement and op- 
erate a program. 

In estimating the Incremental resource requirements, the resources 
available within the district at no additional cost are taken into account. 
These resources could be, for example, assets inherited from discontinued 
programs, physical resources provided cost-free by the coraraunitfy , or vol- 
unteer services. After th^ net Incremental resource requirements are de- 
termined, dlstrl t-speclf Ic resource prices and cost factors are used to 
develop the estimated incremental program cost, usli^g the methodology of 
the planning cost model. Specifically, the standard rtsource costs and 
factors shown in^Flg. 10 are changfid to district-specific costs. 

To illustrate the process and considerations in estimating the incre- 
mental cost of a program, the data for Program E (shown iva estimating the 
comparable replication cost) will l^e used. These data are shuwn in Tables 
6, 7, and 8. 

Data about Program E could have been generated by either the district 
of original implementation or by a state or federal alg^ncy in their evalu- 
ation of programs funded through the agency. Whatever the source, program 
data of this nature is essential information to another district In its as- 
sessment^ of potentially effective **new" programs. 

In this ill^,9trati?n, it is assumed that information about nil the pro- 
grams, A throiyh F, was available and that Program E was tentatively se- ^ 
lected as the moat-likely-to-Bucceed program. Preliminary examination of 
the data used to develop the comparable replication cost (CRC) for Program E 
leads the district planners to believe that the incremental co5t to its 
district will be ijlgnif icantlv lower. The district's current salary ached- 
ule sets average teacher 8alat> at $9000 and paraprof esslonala at $4000. A. 
major portion of the equipmertt and materials required for the program are 
available within the district. 



-184- 



Table 6 

PROGRAM AND RESOURCE INFORMATION FOR PROGIUM E 
An Elementary Level, Reading and Mathemeulcs Program 



Desoiriptors 
Students Served 



Instruction 
Class time 

Number of students 
Students/instructional area 
Number of sections 
Utilization 
Facilities 
Space 



Furnishings 



Staffing 

Certified teachers 
Special teachers 
Paraprof essionals 

Equipment 



Materials 

Pre-servlce Training 
In-service Training 
Other support 



Heaource 1 yifoma /. i on 

Grades 2-4 
Title I; low SKS 
Underachievers 

1.25 hours - Reading 
1.25 hours - Mathematics 
103 

50± 
2 

5 hours /day 

2000 square feet 

1 instructional area 

1 activity area 

6 carrels 
Carpeting 

i Tables and chairs , 

1 per instructional area 
None 

2 per instructional area 
1 per activity area 

Telex (remote diagnos t 1 c) 
Tape recorders 
Cassette players 
Headsets 

Books, games, incentives 
5 days - formal 

3 days - formal 

Remote diagnostic-Prescrip- 
tive services 



Quantity and quality of items would be specified in 
supporting lists. 
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Table 7 

COST INFORMATION FOR PROGRAM E 
(Costs in dollars) 

I ten Cos t 



Facilitie:^; Cos. ^ 

Total program c '^t ^ 7,500 

Cost per instructional area 5,000 

Equipment Cost 

Total ^ 2,000 

Cost per Instructional area 2,000^ 

Number of instructional areas 1 

Students per instructional area 50 

Replacement factor 10% 200 

Maintenance factor 10% 200 

Materials Cost 

Total 8,600 

Cost per instructional area 8,600^ 

Number of instructional areas 1 

Consumables ($ per student) 10 

Pre-service Training ^ 

Number of staff days 20 

Cost per day^ 200 

Total cost 4,000 

In-servicu Training 

Number < f staff days 12 

Cost per day 200 

Total nost 2,400 

Other support ^ 

Student dfagnoiiitic service- 50 

Program evaluation ($ per student) 10 

Consultants ($100 per day) 800 



One instructional plus one activity area. 
^Includes time for paraprofesslonal staff. 
Includes salary, materials^ and training costs. 

d 

Contracted diagnostic and prescriptive services. 
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Table 8 

COMPARABLE REPLICATION COST EOR PROGRAM E 
(In dollars) 

T.f.m Cr/if 

Acqu- * slt lon Cost 

F'^'^. Ill ties (remodel/ furnish) 

'i')tal program cost 7,500 

(Cost per Instructional area) (3,750) 

Equipment 

Total program cost 2,0C 

(Cost per instructional area) (2,00.) 
Materials 

Total program cost 8,600 

(Cost per instructional area) (8,600) 

Pre-service training A ,000 

Total acquisition cost 22,100 

Operational Cost 

Salaries (including fringe benefits) 

Teachers ($12 ,000/year) 12,000 

Paraprofessioniils ($5 ,000/year) 15,000 
Other (variable) 

Maliarials 

Program-related (10%) 860 

Consumables (student-related) 1 ,030 
Equipment 

Replacement (10%) 200 

Maintenance (10%) 200 

In-service training 2 ,A00 

Other support 

Student diagnostic services 5,000 

Program evaluation 1,000 

Consultants 800 

Total operational cost 38,A90 



Diagnostic and prescriptive services by 
contracted services • 
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For this district, the CRC for Progrnm H repres(»nts a maximum ex- 
pected program cost. For another district, with a hifiher salary sched- 
ule and no equipment or materials on hand, the CRC for Program K would 
be lower than its incremental cost. Both districts gain needed insights 
about the cost Impact of Program E frc^m ^ust a quirk look at tho CRC 
for Program These Insights cannot bo developed if the only rnst in- 

formation the district has about Program K is a cost per student or 
tho total program cost specific to the district originally devel opi*^" 
the program. 

In developing the program cost estimates for use In designing the 
scope and nature of Progvam E, the district determines the resources 
available within its inventory and matches this information with the 
resources req»»ired to implemijnt and operate the program. The result- 
ing Incremental resource requirements are translated by meens of tlie 
planning cost rijodel into an estimate of Incremental cost. In this 
translation process, district-specific resource prices and factors are 
used « 

The data needed and the results of the Incremental cost analysis 
for the various configurations of Program E are oresented in the same 
formats as Tables 6, 7, and 8. As an Illustration, the incremental 
cowt for two program configurations (160 students and 200 students) is 
shown in Table 9< The assumptions. Incremental resource requirements 
and district-specific resource prices supporting the cost estimates 
would be displayed, in ppuotioe^ in the formats of Tables 6 and 7. In 
this illustration, most of the information can be identified in Table 9, 
J St briefly, the district has in Inventory about 50 percent of the re- 
quired equipment for a program of 100 students. Adequately remodeled 
space is available for one instructional area and one activity area, 
But I two instructional areas and activity areas are needed for 160 stu- 
dents < Only carrels have to be purchased in order to furnish as many 
as four centers. For one configuration, the district looks at the cost 
impact of developing an In-house capability for the diagnostic-prescrlp-* 
tlve services that are provided to the other configurations on a con- 
tracted basis* This leads to an Increase l.*? the cost uf pre-servlce 
training and the additional operational cost for etaff members to pro- 
vide this prograifl-related service* 
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Table 9 



INCREMENTAL COST ESTIMATES FOR ALTERNATIVE CONFIGURATIONS OF PROGRAM E 

(In dollars) 



Program Cog t Ca tegory 



Ej_ hj_ ^ 

160 students 200 students 260 r.tndents 



Acquisition Cost 

Facilities (Remodel/furnish) 3,500 
. (1 instructional and 1 activity area 
have to be remodeled) 
Equipment 3,000 
(Unit cost/instructional area for 40 
students is $2,000) 
Materials 13,000 
(Unit cost for instructional area 
for 40 students if $6,500) 
Pre-service Training 4 ,000 

(5 days per staff member and 
training of forty days for diag- 
nostic services in E3 

Total Acquisition Cost 23,500 



3,500 
3,800 
17,200 
8.000 

32,500 



3,500 
3,000 
13,000 
12,000 

31,500 



Operational Cost 

Salaries 

Teachers ($9,000)' 

Paraprof essionals ($4,000) 

Other ($5,000/1/3 time) 
Materials 

Program- related 

Consumables 
Kquipment 

Replacement 

Maintenance 
In-service Training 
Other support 

Student diagnostic services 

Program evaluation 

Consultants 

Total Operational Cost 



(2) 18,000 
(2) 8,000 



1,300 
1,600 

400 
400 
3,200 

8,000 
1,600 
800 

43,300 



(.2) 18,000 (2) 18,000 
(6) 24,000 (2) 8,000 



1,720 
2,000 

500 

500 
6,400 

10,900 
2,000 
800 

65,920 



1,300 
1,600 

400 
400 
3,200 



3,200 
800 

41,900 
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The resulting program cost analysis provides the information needed 
by the district in making the decision about whether to plan the iniple- 
mentation of the program and, if so, what configuration of program can 
be afforded within the resource constraints of the district. As a final 
note, two points should be made clear. First, these cost estimates 
are playining cost estimates. Much greater detail and accuracy are re- 
quired to meet the needs of actual implementation and financial accoun- 
tability. -Second, analysis of the dollar-cost alone does not provide 
adequate InfoTnatlon for educational decisions; for this reason the 
emphasis here is on the analysis of both the dollar and non-dollar re- 
sources required for alternative programs. 
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Appendix 

DETAILS OF PROGRAM AND RESOURCE INFORMATION 
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Table 10 



PROGRAM AND RESOURCE INFORMATION 



Program A 



Descynpiops 

Characteristics of StucA^nts 
Served 

Number of Students 

Instructional 

Class time 

Class size 

Facilities 
Space 

Students/classroom/ day 

Utilization 

Furnishings 

Staffing 
Teachers 

Special teachers 
Paraprof ssionals 
Other personnel 

Equipment 

Materials 

Pre^service Training 
In-service Training 
Other Support 
Incentives 



^nfomati on 



Grades 7-12 

Educationally disadvantaged (at least 2 
years below level) 

350 (Reading 
)Math 

^l period Math 
)l period Reading 
20 students per classroom area 

4 trailers @ 900 sq ft 

2 classrooms @ 1000 sq ft 

6 hr/day; three 2-hr shifts 

Desks, carrels, carpet, air condi^-loning 

6 
0 

6 

Project manager; associate manager 
Dorsett M^86 Teaching Machines 
Filmstrips , records 
1 week per teacher 

5 days total 
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lable 11 

PROGRAM AND RESOURCE INFORMATION 
Program B 

Descriptor Information 
Characteristics of Students 7-12 grades 



Served 
Number of Students 



Educationally handicapped (at least 2 years 
below grade level) 

285fReading 
(Math 



Instruction 

Facilities 
Space 

Nunber of students 

Utilization 

Furnishings 

Staffing 
Teachers 
Specialists 
Paraprof essionals 
Other staff 

Equipment 
Materials 

Pre-service Training 
In-service Training 
Other Support 



l1 period Math 
1 period Reading 
,20 students per classroom area 

4 trailers @ 900 sq ft 

1 classroom Q 1000 sq ft 

20 per classroom area 

6 hr/day; three 2-hr shifts 

Desks, carrels, carpeting, air conditioning 



5 
0 
5 

Project manager; associate manager 

EDL, AUD-X, Tach-X, controlled readers, 
Flash-X 

Filmstrips , discs 

40 hr per teacher and aide 

No formal training 

None 
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Table 12 

PROGRAM AND RESOURCE INFORMATION 



Program C 



Deo cTnp tors 



Characteristics of Students 



Instruct lonil 

Number of stuJents (as of 

inid-Dec€'Jiber) 
Class time 
Class size 

Number of seccions 

Facilities 
Space 



Stud'^nts/classrooTi/day 

Furnishings 

Staffing 

Certified teachers 
Special teachers 

Paraprofesslonals 
Other personnel 

Equipment 

Primary unit 
Supplementary system 

Redundant systea 
(laterials (lO'X consumable) 



JnfoT^'ction 



Pr«-«crvice training 

In-servlcc trnlntng 
Other Support 



[Grades 6-9 

ITranslerit Y\{^ ycai^y turnover 

/Black, model cities neighborliood 
^ Lov inconic 

FLowest achievers according to last spring's testing 

I Specialized pupils included 

iProgran pupils distributed among all homerooms 

A91 (Reading); 535 (Math) (same students) 
A5 minutes/day (Reading and Math each) 

35-AO In single center (SC) (AO optimum); i^-hb Iti double 

center (DC) (optimum) 
lA each (7-period day) 

/it centers: I DC for reading and math; 1 SZ for reading 
and 1 SC for math; each center has an Instructional and 
an AM*J arfa 

II relnforcetrent room 

^ total occupies sp.ice of 7 fonne- classrooms (walls Cere changed) 

No. students per da y « (A91 + 535) 
No. classrooms 7 

f Table space fo" carr*»ts 
Carpeting 
Air conditioning 
1 carr':l per student per class (i.e., approximately lAO total)* 
Chairs 

1 per center (Reading and Iiath each) 
None 

(Full-time: 1/center; 1 for reinforcement room 
\l Substitute 

!1 full-tlnc director 
1 full-time secretary 

Reading Math 



1A7 



40 Hoffman Reading machines 
25 tape recorders/center 
(50 total) 



15 Borg-Warner System 80 



40 tape recorders/center (80 total) 
40 fiashcard readers (Electronic 
Futures, mfg.) 



Math mini system (tapes) 
Workbooks (not on per pupil basis) 



(2 sets EPL tapes/center 
2 sets Hoffman naterlnls 
\ (levels B to G)/cenCer 
iWorkbooks (not on per pupil 
V basis) 

2 sets Borg-Warner materials (levels 1-8) per reading and math 
center (i.e., of complete sets) 

1 notebook per student for compLlln^ materials 
fOnc week on AMS in-depth training 
|0nc week going through materials 

About 2 hr/week 

Konc, instructional program self-contaltied 
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Table 13 



PROGRAM AND RESOURCE INFORMATION 
Program D 

Desaviptors Infoma ti oi 



Characteristics of Students 



Program Scope 
Instruction 
Number of students 
Class time 

Class size 

Number of sections 

Facilities 
Space 

Students /classroom/day 
Furnishings 



Staffing 

Certified teachers 
Special teachers 
Paraprofessionals 
Other personnel 

Equipment 
Telex 

Cassette tape records 
Materials 

Pre-service Training 
In-sarvice Training 
Other Support 



[Grades 1-6 

I Inner-city , b lack , 1 ow income 
^Transiency = 30% 

|T,owest achievers for first 5 months, tiien 
entire school (excluding most special 
education students) 

Reading and math 

Initially 100, later 150 (as of February) 
Initially 2-1/4 hr, later reduced to 75 

minutes for grades 1-4 
45-55 (maximum at 60) 

three (one each for grades 1 and 4, 2 and 
3, and 5 and 6) 

Two regular classrooms 
75 

(30 carrels and chairs, with electric out- 
lets at each carr«^;l 
7 tables , 21 chu Irs 
3 bookshelf-cabinets 
iCarpetlng 

One (no outside preparation required) 
None 

Two , 6-hi day 

On-site director and secretary 
1 

30 

BRL modern math texts 

Large variety of other materials 

Five days for entire staff of school 

Eight morning meetings for entire staff 

None 
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Table .14 



PROGRAM AND RESOURCE INFORMATION 
• • Program -E 



Descrtptors 

Characteristics of Students 
Served 

Instruction 
Class time 

Number of students 
Class size 
Number cf sections 
Utilization 

Facilities 
Space 



Furnishings 



S taf f ing 

Certified teachers 
Special teachers 
Paraprof essionals 

Other personnel 
Equipment 



Materials 

Pre-service Training 
In-service Training 
Other Support 
Incentives 



Information 

Grades 2-A ; Title 1 
Low SES 



1.25 Reading 
J .25 Math 

103 

50 students per class 
2 

5 hr. per day 

|2000 sq ft 

classroom 
f 1 activity area 

(6 carrels 
. Carpeting 
jTables 

1 per center 
none 

12 per center 

\l per activity area 

Telex 

Tape recorders 
Cassette players 
Headset 

Books , games , toys 
5 days 

4 days , total 

Remote diagnostic and prescriptive 
25 per s tudent-^candy , scrip 



Table 15 



PROGRAM AND RESOURCE INFORMATION 
Program F 

Descriptors ' Information 



Characteristics of Students 
Served 

Number of Students 

Instructional 
Class time 
Class size 

Number of sections, school 

Facilities 
Space 

Students/ classroom/ day 

Utilization 

Furnishings 



Staffing 
Teachers 

Special teachers 
Paraprof essionals 
Other personnel 

Equipment 



Materials 

Pre-service Training 
In-service Training 
Other Support 
Incentives 



Title I students 
250 

50 minutes 
25 
5 

Regular classrooms 

125 . 

100% 

Air conditioning, pleasant environment 
small, modern (partitions, file cab- 
inets, storage cabinets, etc., loose 
table, chairs) 

1 classroom ■ 

0 

1 

1 program director 

6 Cassette players ($25) 
6 tape recorder ($150) 
Earphones ($50) 

Sound filmstrip sets 
Cassettes 

Workbooks and miscellaneous supplies 
1 week 
3 days 

Evaluation: $10 per child 
300 books given as awards 
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Appendix D ^ 
SCALING OF STUDENT PERFORKANCE 

A number of techniques are described in the literature' on psychological 
scaling for making subjective judgments, and all of them yield a set of met- 
ric values associated with the stimuli (i.e., a paper or other sample of a 
student's performance). These numbers can be treated statistically like any 
other set of numbers since they have at the least, the property of addition 
(interval scale). The nature of the problem considered here makes the method 
of direct estimation appear most promising. Tliis method has been described 
in a number of places by Stevens and Galanter. In direct methods of esti- 
mation, the judge assigns a number to each presented stimulus relative to a 
standard stimulus. For example, given a standard light intensity of value 
10, other intensities are assigned a number relative to the standard. Usually, 
these estimates are ratio estimates; the judge estimates each stimulus as 
being either a fraction or a" multiple of the standard.' Tlie method is not 
restricted to ratio estimation, however, and a ratio method is not recommended 
for use in evaluating student performance. 

In estimating the value of samples of student's work, judges are pro- 
vided two standard stimuli and all others are judged relative to the stan- 
dards. Standards are chosen with scale values of 25 and 75, and judges are 
instructed to rate all papers from 0 to 100, with 0 defined as completely 
worthless and 100 as perfect. The score on any given paper is simply the 
average across judges and th^ reliability of a score is determined by a 
relatively few judges using a categorical scaling method or they would sim- 
ply be picked by "experts." The adequacy of these procedures must be in- 
vestigated. 

All students in 'a program in the same grade would be given a common 
assignment, one that could be repeated at various times in the program. 
For example, fourth-grade students might write a paper on "Wliat I Did Last 
Weekend." The important thing is that all students in a grade be given the 
same assignment; otherwise it would" be impossible to scale papers across 
classes for the whole grade. Depending on the grade level and the competence 

Stevens, S. S., "A Metric for the Social Consensus," Science , 151, 1966, 
pp. 530-541. Galanter, E. H. , "Contemporary Psychophysics ," in New Directions 
O g Psychology , Holt, New York, 1962^ pp. 89-156. 
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of the students, several kinds of schoolwork should be sampled, such as 
writing assignments, arithmetic, social studies or science, and perhaps 
art. The type of work sampled and the specific nature of the assignment 
must be worked out with taachers because the assignments should be typical, 
of the general schoolwork done in a specific school. These special assign- 
ments will not be graded or otherwise marked by the teachers. The stu- 
dents* names will appear on the papers, although they will be removed be- 
fore judging and a code number assigned instead. After the code is assigned 
the papers are given to a panel of judges and each judge assigns a score. 

One of the major difficulties in consistent scoring of this type of 
material is that the stimuli are multidimensional. For example, a written 
paper may have uneven quality in level of content, organization, neatness 
and Spelling. If judges differ in the relative weights they assign to 
various dimensions, poor interjudge reliability will result. While methods 
for multidimensional scaling exist, they are too complex to be feasible in 
the present application. To have all judges scoring along the same dimen- 
sion, careful instructions have to be developed and tested for efficiency. 
Preliminary pilot investigations indicate that high interjudge reliability 
can be obtained and that the procedure eliminates the disagreement between 
judges (teachers) found in usual methods of scoring this kind of student 
performance; i.e., teacher grades . 

The validity of the scaling method is determined primarily from mea- 
sures of interjudge reliability. A related effort investigates the relation 
ship between scaled scores and scores achieved on standardized tests for 
creativity, reading, and mathematics, so as to partially validate both kinds 
of scores. The primary analysis is based on a correlation study of the re- 
lationship between standardized and scaled scores. This analysis not only 
indicates the general agreement between the two scores, but allows for a 
more meaningful diagnosis of the kinds of discrepancies that occur. For 
example, the analysis might reveal that the co^-relation is poor for students 
scoring very low on the standardized test, indicating that low scorers on 
standardized tests tend to perform at a higher level on meaningful high leve 
learning. Again, the correlation might be poor for st.udents who are behav- 
ioral problems, indicating that poor behavior is manifested more on one kind 
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of perfonnance measure than another. The pattern of discrepancies allows 
one to erect explanatory constructs and to attempt to isolate responsible 
variables. Of course, used in this way, the analysis is hypothesis-generating 
and not hypothesis -tes ting. But hypotheses generated by one set of data can 
be tested on anothler., 
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Appendix E 
TH^ DELPHI ATTITUDE ESTIMATION METHOD 

The purpose of this procedure is to determine attitudes about school 
objectives as seen by various members of the education community since an 
obvious source of disparity between the community and the school lies J.n 
the educational objectives each thinks are important. Tlie method to 
be used for arriving at scaled attitudes is a modification of psycho- 
logical scaling procedures and has been developed over several years 
at Rand. This method, generally referred to as Delphi, is becoming 
increasingly popular. Two important features of the technique are 
(1) participants construct attitude statements in their own language, 
and (2) a set of commonly agreed on statements are derived and scaled 
in terms of importance. 

ITie method for obtaining attitudes about educational objectives con- 
sists of two independent operations. One is a procedure for generating 
attitude statements, and the second is a procedure for assigning rela- 
tive values to the statements. The first operation will require two 
sessions (of about one , hour in duration) with approximately 150 partici- 
pants. These two sessions will result in a set of 15 to 20 statements 
about the objectives of education. In order to assure that the set of 
attitude statements cover a wide range, the participants must represent 
the various ethnic and SES subgroups of the school and community. For 
this purpose about 80 to 100 parents should participate and about 40 to 
60 school personnel, making up the total sample size of 150 subjects. 
Attitude statements can then be separated by groups, or combined into a 
set of overall group statements. The latter is preferred because then 
differences between groups can be analyzed primarily in terms of differ- 
ences in the importance assigned to each objective statement. Analysis 

The use of parents in this survey is dependent upon two conditions: 
(1) that low income parents will be ataenable to the technique (prelimi- 
nary evidence indicates this will not be a problem) , (2) that this addi- 
tional survey does not overburden parents who, as a group, are being 
rather extensively sampled and interviewed in the parent/ community sur- 
veys . 
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is discussed in more detail below. It is important to note in the proce- 
dure that follows that it is not necessary to have all subjects together 
at the same time. Tlie second operation, in which participants estimate 
the importance (or value) of each objective statement, requires very 
little of the participants' time. 

In addition to determining subjects attitudes about education objec- 
tives, we will also have them evaluate their school in terms of how well 
it meets each objective. ! 

The basic steps iu the procedure for determining 55chool objectives 
and for evaluating the schools are outlined below. These are expanded 
in the following pages . 

OVERVIEW 

Phase I. Determine School Objectives. 

a. All subjects generate education objective (EG) statements. . 

b. A small (no more than 7 members) group of "experts^* reduces the 
set of items generated by all groups to a list of 100-200 items 
by identifying highly similar items. 

c. All subjects sort the items in the reduced list into 20 or less 
categories on the basis of similarity - of education objectives. 

We will also have them evaluate their school in terms of how well 
it meets each objective. 

d. A hierarchical clustering routine* is employed to generate a 
set of common objectives, based on the sorting data from c. 

This list is expected to contain between 15 and 20 items. 

e. Each subject rates the relative importance of the items on the 
common list using the method of magnitude estimation. 



This is a technique for partitioning statements (or any object) into 
optimally homogeneous groups using empirical measures of similarity between 
statements. The technique merges clusters in a step-wise fashion beginning 
with each statement as a separate cluster to all statements merged into a 
single cluster. Tlie user selects' the level of discrimination which suits 
his purpose. 
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At this point, a set of EO statements have been generated and their 
importance rated within each group. Differences in EO as seen by the 
different groups can then be investigated. 

Phase II. Evaluate Scliools 

The second phase of the evaluation is Lo determine how subjects feel 
about their school in terms of the EO generated by Phase 1. This is 
accomplished by having all subjects rate their school in terms of how 
well they feel the school meets each of the objectives generated by "d". 

T].ie above outline briefly states the steps in the procedure that 
will be carried out in the evaluation. Each of th^'^.se steps will be 
elaborated in detail. 



Detailed Procedure 

In the first session, subjects are instructed to write out three 
to seven statements about what they think EO should be. Tlie following 
is an abbreviated illustration of the instructions to be read to all 
groups of subjects: 

The purpose of this session is to let you state what you 
think the goals of (elementary and high school) education should 
be. That is, what are the most important things that the 
schools should accomplish both for the students and for the 
community. You have been given a set of 4 x 5 cards. On each 
card write one statement saying what you think an important 
goal for the schools should be- Write down at least three 
such statements, but no more than seven. Write only one 
statement on each card. Write down what you think are the 
most important things the schools should accomplish, even if 
you think some of these are not being done, or not being done 
well, by the schools right now. Are there any questions? 

Many of the statements obtained will obviously be like other state-- 
ments, and some will obviously differ from others, with many statements 
that are neither (obviously) alike or different. The next step is to 
reduce the number of statements by pooling those of common meaning. A 
small group of five to seven experts will take the combined list of 
objectives from all oi the subject groups and identify those with highly 

An expert panel will consist of several individuals selected from 
tlie school and community.. 
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similar or identical meanings ► With 150 subjects the original list will 
probably liave between 600 and 800 items ► By identifying the very similar 
items, hopefully a compression to less than 200 items can be accomplished 

The reduced list of statements will be reproduced on 4 x 5 cards and 
all subjects will rate these statements in terms of their common meaning. 
Subjects will be instructed to place the statements into 20 categories 
with similar statements in the same category. Tliey will be -ins true ted to 
place some statements in all categories forcing equal spread across cate- 
gories for all subjects. Each subject will perform this exercise indepen- 
dently so that 150 separate ratings will be obtained. The following 
instructions are read to the subjects' at this step: 

You will be given a set of cards on each of which is a 
statement of an education objective* We would like you to 
place the cards in 20 separate piles, so that all statements 
in the same pile are alike, or nearly alike and statements 
in different piles are different. Some of these statements, 
will be very much like other statements, and others will be 
very different. Sometimes it will be difficult to tell if 
two statements are alike or different. Don't worry about 
being exactly right in these cases, and do the best you can* 

The next step is to combine the individual judgments of item similar- 
Ity into a single set of 15 to 20 statements which represent the "average' 
agreement in the group. The derivation of this final set of objective 
statements is accomplished by the method of hierarchical clustering. 
This completes the procedure for developing a set of attitude statements 
defining education objectives. 

In order to evaluate the relative importance of each statement, the 
subjects will rate statements using the technique of direct or magnitude 
estimation. In this method, the subjects simply assign a number between 
0 and 100 to each statement where 100 is the highest value and 0 is the 
lowest. Subjects will be given a list of the statements, with a space 
marked beside each one for their value estimates. The following instruc- 
tions will be read to the subjects; 
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On the sheet in front of you there are 20 statements 
about education objectives. These 20 have been reduced 
from the original ones that each of you wrote out in the 
first exercise. These statements may not completely cover 
everything that all of you feel are important, but they 
should -cover the ones you felt were the most important, and 
in many cases, the only difference between what you may have 
said and the statement in front of you is a matter of word- 
ing. Wiiat we would like you to do now is to tell us how 
important you think each of these statements is. You will 
note tha-fej beside each statement there is a place for your 
estimate. We would like you to estimate the importance of 
each objective by assigning it a number from 0 to 100, where 
0 is not at all important, and 100 is most * important . Do 
this for each statement so that when you finish, the most 
important statement has the highest score. (Example) 

The result is a set of rated statements about education objectives. 

Evaluation of Gchools 

In the final phase subjects will be asked to rate their school in 
terms of the EO statements. Again, they will use a number from 0 to 100 
to indicate how well . their school meets each objective. Subjects will 
also be asked to give a general estimate of how well they think their 
school is doing. A space on the back of the objective statement sheet 
will be included for this. 

Tlie following instructions will be read to the subjects: 

On the sheet in front of you there are 20 statements about 
education objectives. These were produced by a group selected 
from your school and community. Wliat we would like you to do 
is to rate your school on how well it meets each of these objec- 
tives. If you think your school meets an objective as well as 
it is possible give it a score of 100 on this objective. If 
you think your school does not meet the objective at all, give 
it a score of 0* Use a number between 0 and 100 to represent 
where you think the school stands in meeting each objective. 

When this is finished, subjects will be instructed to turn the page 
and the following instructions will be given: 

Now we would like you to estimate how pleased you are with 
your school in general. Again use a number from 0 to 100, with 
100 meaning you are very pleased. 



Whatever number (15 to 20) are selected from the hierarchical cluste 
ing analysis . 
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Analysls 

The full study described above would furnish a rich body of data fur 
analysis • The basic focus would be on the similarities and differences 
in evaluation between the groups ^ Several investigations can be made 
from the same data base: (1) The differences in perceived objectives 
can be made explicit by generating separate sets of clusters for each 
group, using, just the sortings of that group in the hierarchical cluster- 
ing routine* (2) Differences in perception of the relative importance of 
the common objectives can be generated by computing group importance rat- 
ings for each group separately and subjecting the data to an analysis of 
variance* (3) The usefulness of the common set of objectives for pre- 
dicting the overall degree of satisfaction of the groups with present 
educational systems (separately, or in common) can be examined by com- 
puting the linearly weighted combinations o£ the individual's ratings of 
his school, and comparing this with his overall rating. Another approach 
to the same question could be carried out by computing the estimation 
weights for each of* the objectives in a linear estimation model for over- 
all satisfaction. 
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Appendix F 
TEST MATERIALS 
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COOPERSMITH SELF-ESTEEM INVENTORY 

Please mark each statement in the following way: 

If the statement describes how you usually feel, put a check (/ ) 
in the column, ''Like Me." 

If the statement does not describe, how you usually feel, put a check 
( / ) in the column, "Unlike Me." 

There are no right or wrong answers. 

Like M e Unlike Me 

1. I spend a lot of time daydreaming. 

2. I'm pretty sure of myself. 

3. I often wish I wer,?. someone else. 

4. I'm easy to like. ' 

5. ytj parents and I have a lot of fun togecher. 

6. I never worry about anything. 

7. I find it very hard to talk in front of the 

class. 

8. I wish I were younger. 

9. There are lots of things about myself I'd 

change if I could. 

10. I can make up my mind without too much 

trouble. 

11. I'm a lot of fun to be with. 

12. I get upset easily at home. 

13. I always do the right thing. 

14. I'm proud of my school work» 

15. Someone always has to tell me what to do. 

16. It takes me a long time to get used to any- 
thing new* 

17. I'm often sorry for the things I do. 

18. I'm popular with kids my own age. 

19. My parents usually consider my feelings. 

20. I'm never unhappy. 

21. I'm doing the best work that I can. 
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Like Me Unlike Me 

22. I give in very easily. 

23. I can ujually take care of myself. 

24. I'm pretty happy. 

25. I would rather play with children younger 

than me. 

26. My parents expect too much of me. 

27. I like everyone I know. 

23. I like to be called on in class. 

29. I understand myself. 

30. It's pretty tough to be me. 

31. Things are all mixed up in my life. 

32. Kids usually follow my ideas. 

33. No one pays much attention to me at home. 

34.. I never get scolded. 

35. i*m not doing as well in school as I'd 

like to, 

36. I can make up my mind and stick to it. 

37. I really don't like being a boy girl. 

38. I have a low opinion of myself. 

39. I don't like co be with other people. 

40. There are many times when I'd like to 

leave home. 

41. I'm never shy. 

42. I often feel upset in school. 

43. I often feel ashamed of myself. 

44. I'm not as nice looking as most people. 

45. If I have something to say, I usually 

say it. ' 

46. Kids pick on me very often. 

47. My parents understand me. 

48. I always tell the truth. 

49. My teacher makes me feel I'm not good 

enough. . 

ERIC 
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Like Me Unlike Me 

50. I don*t care what happens t:o me. 

51. I'm a failure. 

52. I get upset easily when I'm scolded. 

53. Most people are better liked than I am. 

54. I usually feel as if my parents are push- 
ing me . 

55. I always know what to say to people. 

56. I often get discouraged in school* , 

57. Things usually don't bother me. 

58. I can't be depended on. 
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SEARS SELF-CONCEPT 



Name Boy _Girl Grade 

Teacher 



Some boys and girls have thought about the things they do and decided that 
the items on these pages were helpful in thinking about themselves. This is a 
chance for you to look at yourself and decide what your strong points are and 
what your weak points are, Tliis is not a test; we expect everyone to have dif- 
ferent answers so be sure your answers show how you think about yourself « 
Your answers are privace and will be kept in confidence. 

Read each item and then answer the question: Compared with other boys and 
girls my age, how do I rate now? 

Find the line under whatever heading indicates your answer, (The words at 
the top show what the lines in each columr .cand for,) Mr.rk an X on that line. 
Now go right ahead. Work as fast as you like. 
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Excellent Very i3etter than 
good most 

1. Being good at sports 

2. Learning things rap- 
idly 

3. Making friends easily 

4. Having nev;, original 

ideas 

5. Getting my school work 
done on time and not 

getting behind 

6. Being able to read well 

7. Being a good size and 

build for my age 

B. Remembering what I've 

learned 

9. Being willing for others 
to have their way some- 
times 

10. SolviufT' problems in ways 

others haven't tried 

11. Being confident, not shy 

nor timid 

12. Knowing how to do math 

13. Being good at things that 

require physical skill 

14. Being a good student 

15 . Being a leader — one to 
get things started with 

my own sex 

16. Thinking up answers to 
problems — answers no one 

else has thought of 

17. Being able to concentrate 

18. Being interested in science; 
learning about things that 

scientists do 
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E?;cellent Very Better thaii OK Not so 

good most good 

19. Being attractive, good 

lool'.ing 

20. Having brains for college 

21. Making other people feel 

at ease 

22. Learning about new things 
even when other people 
aren^t interested — study- 
ing about things on my own ~ 

23. (Jetting a lot of fun out 

of life 

24. Writing creative stories 

and poems — 

25. Being a good athlete 

26. Being able to apply 

what I've learned 

27. Having plenty of friends 

among my own sex 

28. Seeing new ways of think-* 
ing about things and put-- 

ting ideas together 

29. Spending most of my time 
on my work, not goofing 

30. Having good handwriting 

even when I'm hurried 

31. Being not too skinny, not 

too fat 

32. Having brains 

33. Being sensitive to what 

others are feeling 

34* Being able to see things 
in my mind easily when 

I want to — ' 



35. Being able to change things 
when they cjon^t suit me 

ERIC 
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Excellent 



36. Being able to spell 
correctly 

37* Enjoying games and 
sports 

38* Br^ing smart 

39* Being active in 
social affairs 
with my own sex 

40. Being interested in 
new things; excited 
about all there is 
to learn 

41* Well organized; having 
materials ready when 
needed 

42. Learning about people 
around the world and 
being interested in 
them 

43* Having nice features 
(nose, eyes, etc.) 

44. Knowing what to do to 
get the right answer 
to a problem 

45. Being easy to get 
along with 

46. Letting my imagin- 
ation go when I 
want to 

47* Enjoying myself 
in school 

48. Doing well in art 
work, painting, or 
drawing 



Very Better than OK NoL so 

good most good 
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ATTITUDE TOWARDS SELF AND SCHOOL (GRADE L-3) 



MARK THE NOSE OF THE FACES YOU CHOOSE 



How do you feel about growing up and 
getting older? 




How do you feel whan it*s time to get 
up and go to school? 




3. How do you feel when you have a chance 
to learn something new? 




How do you feel when you think about 
going home after school each day? 




How do you feel when the teacher tells 
you to get out your books and begin to 
work? 




6. How do you feel when you think about 
how fast you learn? 





How do you feel when the teacher says 
that she is going to give a test? 





How do you feel about how healthy and 
strong you are? 
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9, How do you feel about how well you 
read? 



10, How do you feel about the way the 
neighbors treat you? 



11, How do you feel about how you look 
and the kind of face you have? 



12, How do you feel abouu the way the 
other children treat you? 



13. How do you feel when you get your 
report card and take it home? 



14. How do you feel about how much you 
know? 



15. How do feel about how well you do 
ari thme tic? 



16, How do you feel when you think about 
next year in school? 



17. How do you feel about the way your 
teacher treats you? 



18. 



How do you feel when the teacher says 
that it's your turn to read out loud 
before the group? 
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STUDENT SURVEY 



(Grades 4-8) 



Circle 

Appropriate. 

Response 



Yes No 

1. Do you like school? 1 2 

2 . Do you need more help from your teacher? 1 2 

3. Do you read books from a library? 1 2 

4. Do you like your school? 1 2 

5. Do you enjoy field trips? 12 

6. Do field trips help you in schoolwork? 1 2 

7. Do you get along better outside of school than in school? 1 2 

8. Would you like to spend more time at school? 1 2 

9. Are you satisfied with the grades on your report card? 1 2 

0. Do you worry about your schoolwork? 1 2 

1. Are you doing better in your schoolwork this year? 1 2 

2. Do you look forward to coming to school each morning? 1 2 

3. Do you talk about school at home? 1 2 

4. Has someone from home ever talked to your teachers? 1 2 

5. Do you get praise at home for good schoolwork? 1 2 

6. Do you think you will graduate from high school? 1 2 

7. Do you hope to go to college? 1 2 

8. Do you talk at home about what kind of job or career you 

will have after you are out of school? 1 2 

9. Do you read more than is required by your schoolwork? 1 2 

0, Do you think your teachers usually expect too much of you? 1 2 

1, Do your teachers think you are doing well in your schoolwork? 1 2 
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22, Do your parents think you are doing well in your schoolwurk? 

23, Do you think you could do well in any school subject if you 
studied hard enough? 

24, Are your lowest grades usually your teacher^ s fault? 

25, Do you think you could do well in any kind of job you choose? 



Circle 
/\ppropriui Le 
Response 



Yes 
1 

1 
1 
1 



No 
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Items for Achievement Motivation Inventory 



No Yes 
1 2 3 A 5 6 



1. li/hen you know there are going to be one or two questions 

on a test from outside reading assignments, do you 
always read all the material? 

2. Do you regard yourself as a more consistent and harder 

worker in your classroom assignments than the typical 
student in your classes? 

3. Have others (not your good friends) thought of you as one 

who "missed some of the fun" because you were so serious? 



4. Do you think your fellow students think of you as a hard 

worker? 

5. Do most of your teachers probably think of you as one of 

their hardest workers even though not necessarily one 
of the brightest? 

6. Do other interests (sports, extra-curricular activities, 

or hobbies) prevent you from obtaining an excellent 
rating or mark for effort in school work? 

7. Do you have a very strong desire to excel academically? 

8. Do you try harder to get on the school honor roll or merit 

list than the average student in your class? 

9. Do you try to do most jobs at least a little better than 

what you think is expected? 

10* Do you tend to give up or delay on uninteresting assign- 
ments? 

11. Which do failures most often tend to do to you? 

(Y) Start you off on some, new interest. 
(N) Spur you to new efforts in the thing at which you 
failed. 

12. Are your friends more likely to consider you as 

(Y) Casual and carefree. 
(N) Responsible. 
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Appendix G 

INFORMATION CATEGORIES AND OUTCOME DIMENSIONS 



INFORMATION CATEGORIES 



OUTCOME DIMENSIONS 



1. Education Results 



Political/ Social 

1. Practitioner assessment of local 
schools 

2. Educatioiial goals of practitioners 

3. Practitioner's opinions about EEVD 
4» Parent participation in education 

of children 

Economic/ Cost 

5. Changes in performance of educa- 
tional raarket 

Educational 

6. C9gnitive achievement 

7. Affective growth 

8. Educational objectives of school 
personnel 

9. Teaching plans and practices 
10, Sociology of the classroom 



2. Attitudes of practitioners 



Political/Social 

1. Practitioner assessment of local 
schools 

2. Educational goals of practitioners 

3. Practitioners* opinions abolit school 
integration 

4. Practitioners' opinions about EEVD 

Educational 

5. Educational objectives of school 
persorvnel 

6. Teaching plans and practices 



3. Programs and processes 
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Political/Social 

1. Educational goals of practitioners 

2. Allocation of decision making 
authority 

3. Administrative organization, 
. practice and behavior 

4. Status perquisites within school 
system 

5. Parent involvement in school-related 
activities 



INFOR^IATION CATEGORIES 



Attributes of new schools 
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OUTCO ME DIMENSIONS 

Ecuuomic/Cos t 

6 . Structural changes in educat ionai 
market place 

7. Behavior changes in educational 
suppliers 

8. Changes in performance of educa- 
tional market 

9. Changes in resource allocation 

10. Changes Ln fiscal flows 

Educational 

11. Educational objectives of school 
personnel 

12. Teaching plans and practices 

13. Sociology of the classroom 

Political/Social 
1* Practitioner assessment of local 
schools 

2. Educational goals of practitioners 
3- Practitioners' opinion about school 
integration 

4 . Administrative organization, practice 
and behavior 

5. Legal and constitutional ramifica-- 
tions 

6. Status perquisites within school 
system 

7. SES distribution of students 

8. Parent opinions on integration 

9. Parent involvement in school- 
related activities 

Economic/Cost 

10. Structural changes in educational 
market place 

11. Behavior changes in educational 
suppliers 

12. Changes in performance of educa- 
tional market 

13. Changes in resource allocation 

14. Changes in fiscal flows 

Educational 

15. Educational objectives of school 
personnel 

16. Teaching plans and practices 

17. Sociology of the classroom 
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INFORMATION CATEGORIES 
5, Distributions of students 



6. Allocation of resources 



7. Financial impacts 



OUTCOME DIMENSIONS 

Political/Social 

1. Practitioners' opinion about school 
integration 

2. Demonstration area relationship to 
outside agencies 

3. Legal and constitutional ramifica- 
tions 

4. SES distribution of students 

5. Parent mobility 

6. Community assessment of local 
schools 

Economic/Cost 

7. Behavior changes in educational 
suppliers 

Educational 

8. Sociology of the classroom 

Political /Social 

1. Focus and scope of political 
authority 

2. Demonstration area relationship to 
outside agencies 

3. Allocation of. decision making 
authority 

4. Legal and constitutional ramifica- 
tions- 

5. Position of professionals in 
cotnmunity 

6. Status perquisites within school 
system 

Economic/Cost 

7. Behavior changes in educational 
suppliers 

8. Changes in resource allocation 

9. Changes in fiscal flows 

Political/Social 

1. Demonstration area relationship to 
outside agencies 

2. Legal and constitutional ramifica- 
tions 

3. Voting behavior 
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Economic/Cost 
Changes in fiscal flows 
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INFORMATION CATEGORIES 



OUTCOME DIMENSIONS 



8. Governance and administration 



1. 
2. 
3. 



7. 



9. 

10. 

11. 
12. 
13. 
14. 



Political /Social 
Educationaj goals of practitioners 
Practitioners^ opinions about EEVD 
Focus and scope of political 
authority 

Demonstration area relationship to 

outside agepcies 

Allocation of decision making 

authority 

Administrative organization, 
practice and behavior 
Legal and constitutional ramifica- 
tions 

Position of professionals in 
coTTimunity 

Legal and constitutional ramifica- 
tions 

Parent involvement in ^ chool-related 
activities 

Inter-group conflict and cooperation 
Political and social participation 
Voting behavior 
Political mobilization 



Economic/Cost 
15. Behavior changes in educational 
suppliers 



Political/ Social 

9. Status of professionals 1. Practitioner assessment of local 

schools 

2. Educat'Lonal goals of practitioners 

3. Practitioners* opinions about EEVD 

4. Focus and scope of political 
authority 

5. Allocation of decision making 
authority 

6. Administrative organization, 
practice and behavior 

7. Legal and constitutional ramifica- 
tions 

8. Position of professionals in 
community 

9. Status perquisites within school 
system 

10 » Parent involvement in EEVD options 
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INFORMATION CATEGORIES 



OUTCOME DIMENSIONS 



10. Parent attitudes and responses 1. 

2. 



3. 
4. 

5. 
6, 



8. 
9. 
10. 



Political/Social 
SES distribution of students 
Parent judgments of educational 
opportunities 

Parent assessment of local schools 
Parent opinions on integration 
Parent assessment of EEVD 
Parent participation in education 
of children 

Parent involvement in school-related 

activities 

Parent mobility 

Parent involvement in EEVD options 
Community assessment of local schools 



11. Community attitude and 
responses 



Political/Social 

1. Community attitudes toward education 

2. Community opinions on integration 

3 . Community assessment of EEVD 

4. Community attitudes to political activism 

5. Inter-group conflict and cooperation 

6. Political and fiocial participation 

7. Political mobilization 

8. Voting behavior 



12. Consequences beyond 
demonstration area 



Political/Social 
1 . Practitioners ' opinions about EEVD 
2* Focus and scope of political authority 

3. Demonstration area relationship to 
outside agencies 

4. Legal and constitutional ramifica- 
tions 

5. Position of professionals in com- 
munity 

6. SES distribution of students 

7 . Parent assessment of EEVD 

8. Parent mobility 

9 . Community assessment of local schools 

10. Community attitudes toward education 

11. Community opinions on integration 

12 . Community assessment of EEVD 

13. Community attitudes to political 
activism 

14 . Inter-group conflict and cooperation 

15. Political mobilization 

16. Voting behavior 
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Appendix 11 



SCHEDULE OF WORK AND PROFESSIONAL STAFF TIME R1':QU] REMENTS 

The following figures show the time scheduling of tlie four major 
work elements of the evaluation task, as set forth in text Table V~l . 
They do not show all work elements to be included in Phase II of the 
Analysis and Survey Contract, because a number of them are either con- 
tinuous (general management, analysis of policy implication) or avail- 
able on call (technical supports services for matiiematics and statistics, 
consulting or educational test and measurement problems) . 

The figures are as follows: 



Figure 



Title 



A-1 



Political/Social Analysis 
Component Tasks 

Economic and Resource Analysis 
Component Tasks 

Educational Analysis Component Tasks 

Information Flow Component Tasks 



A-2 



A-3 



A-4 



Each figure shows time period by task, and Rand professional staff 
time requirements by component, during the pre-demonstration and demon- 
stration periods. 
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Appendix J 
BIOGR APHIHS OF KANU Sl'AFF F OR KEVi) 

Senior Rand project sL;jff wi]l include political scitmtists, edn- 
cators , sociologists, economists, psycliologists , statisticians, infor- 
mation specialists, and cost analysts. 

This section provides biographical data on the senior staff: 

Polly Carpenter, education 

Stephen Carroll, economics 

Theodore Donaldson, psychology 

John Farquhar, information science 

Sue Haggart , cost analysis 

George Hall, economics 

Milbrey McLaughlin, education 

Anthony Pascal, economics 

John Pincus , economics 

Marjorie Rapp, educational psychology 

John Rolph, statistics 

Daniel Weiler, political science 

Barbara Williams , sociology 

EEVD assignments for senior staff are discussed in Chapter V. The 
current plan is as follows: 

Project director — Pincus 
Project deputies — Hall, Weiler 

Education task group — Donaldson, Rapp, McLaughlin, Carpenter 

Political/Social task group — Weiler, Williams, McLaughlin » 

Pascal 

Economic/Cost task group — Hall, Haggart, Pascal, Carroll 
Technical support, information system — Farquhar 
Technical support, statistics — Rolph 
Liaison assignments: 
DMC ~ Farquhar 

Field Research Corporation — - Williams 
OEO ~ Pincus, Hall 
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EVA — to be assigned for each site 
ISC — Williams and site director 



ERIC 



